Your GenAI Colleague Is Profiling You
There's More To It Than Meets the Eye (August 13, 2026)
Oh, yes! I quickly became a really big fan of Generative AI. Google Gemini and ChatAI were rapidly becoming my best friends. We worked on all sorts of things ... issue after issue ... from insoluble computer network problems, to how to resurrect a wet iPhone, to why the cat keeps puking on the carpet. It all got very conversational. Interacting with my GenAI buddies was like having a really smart person looking over my shoulder advising me as I did the hands-on stuff.
I quickly noticed that in addition to knowing everything, they also remembered everything. Every once in a while, while working on a new problem or project, they'd bring up something from one of our previous sessions, or from our previous interactions in general. They began to say things like, "From our previous work together, I understand why you think ...", or, "Recalling what you said about (whatever), I think you ..."
At first, I thought that was quite impressive. But then I began to realize that this machine was collecting a lot of personal information about me, and inferring even more on the basis of what it "knew". I wondered if ChatGPT would acknowledge this, so I asked ...
How would respond to this: From my experience with GenAI, I've come to realize that users should understand that, whatever else is going on, the system is also profiling them, and the more extensively it is used, the more comprehensive that profile becomes. As it grows, it begins to affect the responses the system provides. This also raises a privacy concern: who might eventually gain access to that profile information, and for what reason? The course (viz. OpenAI Academy) seems to hint at this, but without addressing it directly.
ChatGPT actually agreed this is a "legitimate issue" (their words), although instead of profiling they call it "personalization", claiming that it enables the service to provide increasingly relevant responses to the user. It further claimed that ...
"ChatGPT currently provides memory and personalization features that can use information from previous conversations to make subsequent responses more relevant. Those features are subject to user controls, and OpenAI provides settings for reviewing or deleting memories, turning memory off, and using Temporary Chat.
I think I was vaguely aware of there being such a thing as "temporary chat" on ChatGPT, but I was not aware that parts of previous conversations could be deleted. Only then did notice the small "Edit Message" pencil icon under my message blocks, which can be used to edit the message, or erase it entirely. Presumably, that also erases the responses. But, of course, we just have to take it on faith that it's really gone. If that's true, what's the point for ChatGPT; how would it learn anything from our conversations?
Curiosity getting the better of me, I then asked ...
Tell me where I live, in what kind of housing, whether I'm married or single, if we have pets, and if so, what kind.
It answered ...
From what you've told me in this conversation and the available context, you live in Michigan, in the Grand Haven area. I don't have your street address, and I wouldn't infer one. You live in an older house, more than 120 years old. You appear to be married; you've referred to "my wife" in discussing the household. You have two cats. That's all I can substantiate. I don't know the exact location of your house, its architectural style, or the cats' names.
In a playful mood, I decided to take this a little further ...
The cats' names are Luna and Iris. Luna is all black, like a panther. Iris is a "tiger" but with white socks (paws). They both belong to our youngest daughter, who rescued them. The house looks like this (attaching a Google street view of our house) ...
The response (edited for length) was ...
Ah, that gives me a much better picture of the situation—and it also explains why your description of the house as "120+ years old" is important.
The house appears to be a small, older two-story wood-frame house with a steep front gable, attached or joined to a later one-story brick commercial/office-style addition. That combination is particularly interesting from the (problem that was discussed earlier) standpoint because the junction between the original house and the later addition gives us another entire category of potential concealed spaces and entry routes.
A few things stand out visually: (a six-point description of the structure then followed, with a discussion about its vulnerability regarding the above-referenced problem).
And now I know the names of your two (cats): Luna, the completely black "panther," and Iris, the tabby with white socks. (smile emoji) ...
... And I'll be careful about the address you just provided: I won't repeat or retain the precise street address as a personal detail.
This last comment was probably meant as a rib, given my early challenge about it's accumulating personal information. Notice also how it gently corrected my description of Iris the cat, referring to her as a "tabby". You'll also notice that I hadn't given it the street address. It took that, no doubt, from the filename of the image that I provided ... "1320FultonAve.jpg" ... and the large brass numbers above the front door of the attached commercial addition.
Continuing the game, I asked ...
Would guess that we use an antenna for television reception, or if we're a cable subscriber.
Its answer ...
That's actually one of the easier guesses from the photograph. (smiley face). I'd guess you use an over-the-air television antenna, rather than being primarily a cable-TV subscriber. The giveaway is the Yagi-style antenna visible on the roof behind the front gable.
So if I were playing the "deduce things from the photograph" game, I'd say probably an antenna.
And given what you've told me about your old-house/old-technology proclivities, I have to admit that an antenna on a 120-year-old house feels entirely plausible. (smiley face)
So, enough of this. I'm sure you get the picture.
As a rule-of-thumb, it's probably "good practice" to treat a Generative AI service like a gossipy old woman who fishes for information and is likely to repeat everything she hears. Sensitive personal information may be "remembered." Beyond that, it will likely "read between the lines" to intuit other probable "facts" about you. If all that is used only to deepen its relationship with you, enabling it to provide responses that are increasingly valid and useful, where's the harm? But what assurance is the that that will be the case? In this age of government intrusiveness, that can't be guaranteed.