This is an interesting question. My recollection is that the CUSTOM flag used to be documented as something that was intended for future use. In general that implies this is not a fully implemented behavior in the client code. There is no CUSTOM variable in the default FILE class, so it isn't clear how it could be used. I tried adding a new variable named "CUSTOM" but I was unable to get any method configured there to execute during the file audit.
There is an existing METHOD variable in the class named ZINIT. I found that if you provide a rexx script to do the file rename and call it from the ZINIT attribute, it will execute, but only one time, when the service is first installed. If the file appears later, after the service has already been run once, then the method is not called at that point. But it might be useful for a one-time requirement to rename the file if present today. For testing I used a rexx script with two input parameters (one for the existing name and one for the desired name). This is a simple two line script named rename.rex:
parse arg oldname newname
call edmcmd 'hide cmd /c ren '||oldname||' '||newname
In the ZINIT method I call the script providing the desired file name. For example, if a file named C:\Temp\test.fil exists, in order to rename it test.adt I configure ZINIT as follows:
rename.rex c:\temp\test.fil test.adt
This is not the ideal solution since it will not detect and rename the file if it appears after the service is initially installed, but it is the best I was able to achieve using file auditing. A more robust solution is a software service that looks for and renames the file. The same script above added as the ZVERIFY method of a software service (that can also deliver the rexx script to machines) provides a solution that will continue to look for and rename the file if it reappears at any future date. If you're interested in that approach, let me know and I will describe it fully.