
The user doesn’t really own the files that appear in share with me. These are files that have been shared with the user they have access may have access to read or write to the file in question but they can not delete it as they do not own the file. That is why the following command will not work
service.Files.Delete(fileId).Execute();
In order to remove the file you must simply remove the permissions on the file granting the user access. The user does have access to remove the permissions on the file.
The Code
Note: This does not appear to work in all cases. I have a file on my Google drive that was shared with me by what appears to be a service account. I have no permissions on the file there for i cant remove my access. I am still digging.
Conclusion
It is possible to remove files from the share with me directory on Google drive. We simply remove our permissions on the file. I would like to thank KrishnaKanhaiya for asking this question on the site here and then posting it directly up on StackOver flow. This was a very good question and not something I have thought of trying before.