> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.
replyNot sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
Not an expert, so I could be wrong, but my understanding is that you could just copy those bytes directly from the wire to your memory and treat those as the Arrow payload you're expecting it to be.
replyYou still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.
If you are in control of two processes on a single machine instance, you could share the memory between a writer and a read-only consumer.
replyThe key phrase though would seem to be “memory representation”m and not “same memory”. You can spit the in-memory representation out to an Arrow file or an Arrow stream, take it in, and it’s in the same memory layout in the other program. That’s kind of the point of Arrow. It’s a standard memory layout available across applications and even across languages, which can be really convenient.
Arrow supports zero-copy data sharing - checkout the Arrow IPC format and Arrow Flight.
replyThanks! This is all probably me using the familiar sqlite hammer where I really shouldn't.
reply