{"_id":"58223619f5bbf21b00e6545f","parentDoc":null,"user":"56157c37c41bfc0d00292b38","version":{"_id":"5429beef1163360800ed31fd","__v":22,"project":"5429beef1163360800ed31fa","createdAt":"2014-09-29T20:19:59.904Z","releaseDate":"2014-09-29T20:19:59.904Z","categories":["5429beef1163360800ed31fe","55e74e1f5d36b32b002563fa","55e7500a73169617001f2e86","55e76b8196131b2f00bf4b72","55e89569cdbb8a350096df9c","55f8935eb089b71700a8364e","55f8a67d3bb4bb0d0022d093","55f9dbefc7ef770d00d9bf6d","5601c21a22ecf60d002b27b3","5601c21ff12aee0d001bb06e","5601c2256c7b610d00280db5","5601c22c22ecf60d002b27b4","5603738d0c78b00d0039af47","5626f70bfcbbc621004ebf52","566f7a1ae144ab0d00e9732f","566f85ca7820960d00c3b93d","56708ce43a32d20d00c45cbf","56708d436995210d003aad8e","5736471abe10a9200030732c","57a82c7aff9bd30e00a6f1c1","591b8500e3992319007190f4","5a3718305c66a60012d960ba"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"__v":0,"githubsync":"","category":{"_id":"56708ce43a32d20d00c45cbf","version":"5429beef1163360800ed31fd","__v":0,"project":"5429beef1163360800ed31fa","pages":[],"sync":{"url":"","isSync":false},"reference":true,"createdAt":"2015-12-15T21:57:56.034Z","from_sync":false,"order":6,"slug":"reporting","title":"Data & Reports"},"project":"5429beef1163360800ed31fa","updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-11-08T20:31:21.264Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":true,"order":12,"body":"To support analytical data needs of **advanced publisher** networks, we have established a Data Log Shipping program to send raw event data files directly to the publisher's data storage of choice, on an hourly basis.  To enable raw log shipping, please reach out to your account manager for exact details.\n\nOnce your account has log shipping enabled, the following process will help set up our data pipeline to start exporting log files to your storage of choice.\n\n##Step 1: Provide storage location, file path format, and access credentials\n\nExample:\n\nS3 Bucket/Path:  `s3://your-data-bucket/YYYY-MM-DD/HH/`\nAccess Key ID:  `ABCDEFGHI123JKLM1NOP `\nSecret Access Key:  `abcdefgHi12JKLmNo+P3RSTXyzABCDeFghiJ4K5`\n\n##Step 2: Data Validation\n\nPolymorph will send a small sample file to your storage location to verify access credential, as well as for your data team to examine the data format and start integrating with your own ETL pipeline and/or business intelligence tools.\n\nSession logs will be collected and merged based on session_id. Fraud validation will be done on the merged logs. \n\n**Log Format:** *Our log files are sent hourly as **multi-part gzipped JSON** files.  Each line of the file contains the complete set of end-to-end events for any ad session on your inventory during the given hour. The event log contains a set of raw **dimensions** for each ad session (e.g. geography, device, placement identifiers, creative identifiers, request time, etc.), as well as a set of **metrics** logged (e.g. request, impression, click, revenue)*\n\nSample log data: \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"uuid\\\": \\\"9era1abc-101f-4fda-93cc-52bd510ceb3a\\\",\\n  \\\"session_id\\\": \\\"12a34567890b1234cde56789f1g23h4i_5678jk9l\\\",\\n  \\\"ip_address\\\": \\\"00.000.00.00\\\",\\n  \\\"geo_timezone\\\": \\\"Europe/London\\\",\\n  \\\"geo_dma_code\\\": 0,\\n  \\\"geo_country_code2\\\": \\\"GB\\\",\\n  \\\"geo_continent_code\\\": \\\"EU\\\",\\n  \\\"ua_device_type\\\": \\\"MOB\\\",\\n  \\\"ua_device\\\": \\\"BlackBerry\\\",\\n  \\\"ua_os_name\\\": \\\"BlackBerry OS\\\",\\n  \\\"ua_name\\\": \\\"Opera Mini\\\",\\n  \\\"ua_patch\\\": \\\"35667\\\",\\n  \\\"url\\\": \\\"http://example.com/page/\\\",\\n  \\\"pub_network_id\\\": 123,\\n  \\\"site_id\\\": 12345,\\n  \\\"zone_id\\\": 23456,\\n  \\\"ad_network_id\\\": 123,\\n  \\\"advertiser_id\\\": 1234,\\n  \\\"campaign_id\\\": 2345,\\n  \\\"creative_id\\\": 67895,\\n  \\\"token\\\": \\\"ABCDEFG\\\",\\n  \\\"r_timestamp\\\": \\\"2016-01-01T16:59:15.550445Z\\\",\\n  \\\"r_cnt\\\": 1,\\n  \\\"r_num_ads_requested\\\": 1,\\n  \\\"i_cnt\\\": 1,\\n\\t\\\"vi_cnt\\\": 1,\\n  \\\"c_cnt\\\": 1,\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n\n**Note** This is just an example of the log format.  Actual dataset is discussed during integration time.\n\n##Step 3: Log Shipping\n\nOnce both data storage and format is fully validated, we will enable our data pipeline to automatically start shipping hourly data to your storage location. Note: there will be delay between event time and shipping time, details will be explained by your account manager.","excerpt":"","slug":"data-log-shipping","type":"basic","title":"Data Log Shipping"}
To support analytical data needs of **advanced publisher** networks, we have established a Data Log Shipping program to send raw event data files directly to the publisher's data storage of choice, on an hourly basis. To enable raw log shipping, please reach out to your account manager for exact details. Once your account has log shipping enabled, the following process will help set up our data pipeline to start exporting log files to your storage of choice. ##Step 1: Provide storage location, file path format, and access credentials Example: S3 Bucket/Path: `s3://your-data-bucket/YYYY-MM-DD/HH/` Access Key ID: `ABCDEFGHI123JKLM1NOP ` Secret Access Key: `abcdefgHi12JKLmNo+P3RSTXyzABCDeFghiJ4K5` ##Step 2: Data Validation Polymorph will send a small sample file to your storage location to verify access credential, as well as for your data team to examine the data format and start integrating with your own ETL pipeline and/or business intelligence tools. Session logs will be collected and merged based on session_id. Fraud validation will be done on the merged logs. **Log Format:** *Our log files are sent hourly as **multi-part gzipped JSON** files. Each line of the file contains the complete set of end-to-end events for any ad session on your inventory during the given hour. The event log contains a set of raw **dimensions** for each ad session (e.g. geography, device, placement identifiers, creative identifiers, request time, etc.), as well as a set of **metrics** logged (e.g. request, impression, click, revenue)* Sample log data: [block:code] { "codes": [ { "code": "{\n \"uuid\": \"9era1abc-101f-4fda-93cc-52bd510ceb3a\",\n \"session_id\": \"12a34567890b1234cde56789f1g23h4i_5678jk9l\",\n \"ip_address\": \"00.000.00.00\",\n \"geo_timezone\": \"Europe/London\",\n \"geo_dma_code\": 0,\n \"geo_country_code2\": \"GB\",\n \"geo_continent_code\": \"EU\",\n \"ua_device_type\": \"MOB\",\n \"ua_device\": \"BlackBerry\",\n \"ua_os_name\": \"BlackBerry OS\",\n \"ua_name\": \"Opera Mini\",\n \"ua_patch\": \"35667\",\n \"url\": \"http://example.com/page/\",\n \"pub_network_id\": 123,\n \"site_id\": 12345,\n \"zone_id\": 23456,\n \"ad_network_id\": 123,\n \"advertiser_id\": 1234,\n \"campaign_id\": 2345,\n \"creative_id\": 67895,\n \"token\": \"ABCDEFG\",\n \"r_timestamp\": \"2016-01-01T16:59:15.550445Z\",\n \"r_cnt\": 1,\n \"r_num_ads_requested\": 1,\n \"i_cnt\": 1,\n\t\"vi_cnt\": 1,\n \"c_cnt\": 1,\n}", "language": "json" } ] } [/block] **Note** This is just an example of the log format. Actual dataset is discussed during integration time. ##Step 3: Log Shipping Once both data storage and format is fully validated, we will enable our data pipeline to automatically start shipping hourly data to your storage location. Note: there will be delay between event time and shipping time, details will be explained by your account manager.