Fitarwa zuwa ma’ajiyar abu

1. Gibin da wannan yake cikewa

Har yanzu akwai hanya ɗaya daidai daga MousePlayer zuwa data warehouse, kuma tana kaiwa ga Google. Idan ana amfani da Snowflake, Redshift ko ClickHouse, babu wata - duk da cewa duka ukun suna karantawa daga ma’ajiyar abu ba tare da matsala ba.

Don haka gadar tana rubutawa zuwa ma’ajiyar abu maimakon kowanne warehouse dabam. Snowflake yana shigowa daga S3, Redshift yana shigowa daga S3, ClickHouse yana shigowa daga S3, kuma BigQuery yana shigowa daga Google Cloud Storage. Fitarwa guda tana isa ga duka.

Wuri	Yadda ake ba da adireshi
Amazon S3	Signature Version 4, bucket a cikin host
Cloudflare R2	Signature Version 4 zuwa wurin ƙarshen asusun
Google Cloud Storage	ta API mai dacewa da S3, da mabuɗin HMAC
MinIO, Wasabi, Backblaze B2 da sauransu	Signature Version 4 zuwa wurin ƙarshen da aka bayar
Azure Blob Storage	hanyar SAS ta container - babu sa hannu a ɓangarenmu kwata-kwata

2. Dalilin bambancin Azure, da dalilin da hakan ya fi kyau

Azure Blob ba ya amfani da Signature Version 4; yana sa hannu da SharedKey. Za mu iya gina tsarin sa hannu na biyu. Maimakon haka gadar tana ɗaukar hanyar SAS ta container - URL da ake samarwa a asusun ma’ajiya na wurin, da sa hannun da yake cikinsa tuni.

Wannan ƙarancin lamba ne a ɓangarenmu da ƙarancin dogaro a ɓangaren mai amfani. Ba mu taɓa ganin mabuɗin asusu ba: abin da ake bayarwa izini ne da aka iyakance da kai, ga container guda, da ranar ƙarewa da aka zaɓa, wanda ake iya sokewa a kowane lokaci ba tare da taɓa wani abu ba.

3. Object guda ga kowace manufa, kuma mabuɗi kwanan wata ne

Ma’ajiyar abu ba ta da aikin ƙarawa zuwa ƙarshe. Fayil na kowace rana zai buƙaci a sake rubuta shi gaba ɗaya a kowane lokaci, kuma manufofi biyu da suka iso lokaci ɗaya za su rubuta a kan juna. Don haka kowace manufa da aka cika tana zama object nata.

Farashin hakan shi ne ƙananan objects da yawa; tsarin mabuɗi shi ne yake sa hakan ya yiwu:

prefix/YYYY/MM/DD/channel/session-goal.json

Warehouse yana karanta rana guda a matsayin partition guda, kuma cike bayanan baya binciken prefix ne maimakon buƙata. Idan ana son raba ta aiki maimakon kwanan wata, a sa channel ID cikin prefix.

Kowane object layi guda ne na JSON

Da gangan babu shigar ciki, don haka ana karanta cikakken prefix a matsayin NDJSON ba tare da buɗe objects ɗaya-ɗaya ba. Timestamp, aiki, zama, ID na abokin ciniki, manufa, ƙima, kuɗi da hanyar sake kunna rikodi.

Ƙima marar komai null ce, ba sifili ba

Manufa ba tare da ƙimar kuɗi ba tana rubuta null. A warehouse sifili zai shiga jimlar da ba ta taɓa kasancewa ba - bambancin “ba a amince da adadi ba” da “adadin sifili” yana tsira daga fitarwa.

4. Idan yana amsa 403, a fara karanta wannan

Buƙatar S3 da aka ƙi ba ta faɗin kusan komai, kuma akwai dalilin da saƙon kuskuren kundinmu yake faɗin fiye da S3: yanki yana shiga cikin sa hannu. Yanki mara kyau da mabuɗi mara kyau suna samar da 403 ɗaya, don haka zargin farko - mabuɗi - yana zama gaskiya a rabin lokuta kawai.

Abubuwa uku da za a duba, a wannan jeri:

• Yanki, daidai yadda bucket yake bayyana shi. R2 da Google Cloud Storage suna ɗaukar auto ba sa duba shi; duk sauran suna dubawa.
• Izinin prefix. Mabuɗin da aka iyakance ga mouseplayer/ ba ya iya rubutawa zuwa mouseplayer-goals/ - slash na ƙarshe yana da muhimmanci a resource ARN.
• Wurin ƙarshen, don R2 da ma’ajiyoyi masu dacewa da S3. A manna host ba tare da https://; muna cire shi ko ta yaya, amma hanyar da aka ƙara a bayansa bucket dabam ce.