A small number of inputs in a large dataset can poison training data pretty drastically. Anthropic wrote a good article about it a while back [0]. This should mean its possible to pull back that information fairly easily.
replyIt is hard to not feed it "secrets" too. Models will see path names, read compose files, etc. Of course you can configure things to not leak this type of information, but its not default in most harnesses and isn't 100% sufficient anyways.
If my experience with image generation is any indication, unless AWS keys are somehow extremely prevalent in the training data, you may get something that looks like one, but it definitely won't be valid.
replydoesn't mean the raw text goes into training. they most likely have a pipeline to clean out any secrets before they train on it?
replyIn theory, but in practice how difficult is that?
replynot hard for secrets with explicit patterns and existing pipelines to detect them
replyUnless they're base64-encoded or compressed?
reply