ntfs file streams. Alternative NTFS data streams, or why the PowerShell script did not start
The article was written for Hacker magazine in 2004. It was published in issue 09/04 (69) under the title “Destructive Currents”.
When you take over another NT system and install your own homemade spyware into it, you need to solve the problem of storing the collected information on the victim’s computer. Usually the log is written to a simple file in a directory with a large number of files, for example, in system32.
NTFS Features
This is a common, but far from the best way to hide information on your local computer. There is a chance that the user will notice an extra, constantly updated file that suddenly appears in his system directory. Append a log to an already existing file? First you need to find a file to which adding information will not spoil its contents. How about saving the information in a place that will not be visible either from Explorer, or from the command line, or from any file manager? The NTFS file system provides us with this opportunity. You rarely see it on a regular home PC, since most users still prefer FAT32, even those running XP. But on the local network of a company running Win2k/XP, NTFS is almost certainly used, because this file system provides such capabilities as assigning access rights to users, encryption and file compression. Additionally, NTFS is much more secure than FAT32. So the data hiding method that I will describe is ideal for industrial espionage. With the advent of Longhorn, NTFS has a chance to find a home on home computer drives, as the upcoming WinFS file system, based on NTFS, promises additional features on organizing and retrieving information that should attract ordinary users.
Attach any data to the file
The method is to save data not to a file, as usual, but to an NTFS file stream. The stream can be attached to another file (in this case, its size does not change, and the data remains untouched, which means that utilities that check file checksums will not notice the changes), to a directory, or to a disk. NTFS Alternate File Streams is one of the features of NTFS that has been present in it since its earliest days. earlier versions Windows NT. It lies in the fact that one file can have several threads containing data, and only the main thread in which the contents of the file is stored is available to the user. There is something similar in the HFS file system on Macintoshes. There, streams are called forks. Until recently, they were used as file resource storage or contained information about the file type. With the advent of MacOS X, Apple recommended placing resources in separate files, and file types are determined by extensions. But branching support still remains. IN Windows threads usually used to store some additional information about a file. For example, a stream might contain a summary of a document. If the system is on an NTFS disk, then the explorer.exe file probably contains a summary. Depending on the summary content, streams named SummaryInformation, DocumentSummaryInformation, and several others may be attached to the file. On my computer I found a thread named $MountMgrRemoteDatabase attached to the C drive.
The user can find out about streams attached to a file only in some cases, for example, when copying a file with an attached stream to a disk with FAT/FAT32. These file systems do not support them, so the system will prompt you to confirm the loss of information in the streams, indicating their names. Of course, this situation will never arise if the stream is attached to a disk or to a system folder. It is not necessary to use threads for spying purposes. If you are a developer shareware programs, then you can easily use streams to store information about registration, the number of days until the expiration date, in a word, everything that should be hidden from the user of your program.
Working with Threads
There are both similarities and differences in working with files and streams. There's not much like it. Both files and their streams are created and deleted by the same WinAPI functions CreateFile and DeleteFile. Reading and writing are implemented, respectively, by the ReadFile and WriteFile functions. This is where the similarities end, only differences follow. Thread names may contain special characters that cannot be part of the name. normal file: such as "*", "?", "<”, “>" ,“|" and a quote character. In general, any stream name is saved in Unicode format. Service characters from the range 0x01 – 0x20 can also be used. There is no standard function for copying and moving a stream: MoveFile and CopyFile do not work with streams. But no one bothers you to write your own functions. Streams do not have their own attributes, creation or access dates. They are inherited from the file they are attached to. If the file itself contains any data, then it can also be represented as a stream. Stream names are displayed as "filename:streamname:attribute". The standard attribute of the stream in which the data resides is called $Data. There are many other attributes whose names also begin with the “$” sign. The contents of the file are in an unnamed stream (filename::$DATA). With this property file system representing the contents of a file as a stream was related to a bug in the old Microsoft versions IIS, when a hacker who wanted to find out the text of a script on a vulnerable server simply added “::$DATA” to its name, and the server, instead of executing the script, gave out its source code. Working with streams is similar to working with files. Take a look at Listing 1. This is a simple example of a program that creates a stream file and writes information to it. After running the program, an empty “testfile” file will appear in its directory. You can see the contents of the attached stream by typing “more” in the command line< testfile:stream». Как видишь, имя потока указывается после имени файла, отделенное от него знаком двоеточия. Самое трудное при работе с потоками – это получить их список для конкретного файла. Стандартной функции нет, и поэтому придется писать ее самому. Напишем небольшую console program, which would return a list of threads by file name. The guys from Sysinternals have such a program, with open source, and it works, but I didn't like their way. They use Native API calls and therefore their code is large and difficult to understand. We will write our own program, which will work from the command line, with a simpler algorithm and standard API functions.
Getting a list of threads
The algorithm is based on the use of the BackupRead function. It is intended for Reserve copy files. When you do backup copy file, it is important to save as much data as possible, including file streams. The information is taken from the WIN32_STREAM_ID structure. From there you can get the stream name, its type and size. We will only need streams of type BACKUP_ALTERNATE_DATA. All functions and structures are described in the winnt.h header file. First you need to open the file for reading using CreateFile. In the dwFlagsAndAttributes parameter, you must specify the FILE_FLAG_BACKUP_SEMANTICS flag, which will allow you to open not only files, but also directories. Then we launch while loop, which reads information about the file into the sid structure, from which we will get information about each thread. Before the next pass of the loop, we clear the structure and move the file pointer to the next stream using the BackupSeek function. After all the threads are found, we clear the lpContext containing service information and close the file. The source code of the program is shown in Listing 2. You can take the already compiled program from our disk. To work with streams, it is not necessary to write special programs. There are some things you can do directly from the command line. Some examples are shown in the box.
Detection
Once a stream of information is attached to something, its contents are difficult to access without knowing its name. If a stream is attached to a logical volume, then in Windows there is no standard means to discover it. Since the stream name may contain characters that are not allowed in regular file names, this creates additional difficulties when trying to find out the contents of the stream using command line. The document summary content is typically stored in a stream with a name that contains the character code 0x05. This character can be typed in the console (Ctrl+E), but if it were the character 0x10 or 0x13 (carriage return and line feed), then it would be impossible to type them. Theoretically, you can find out about attached threads by chance, using some software that is likely to be on your computer. There is an option in WinRAR, and if it is enabled, you may notice that the size small file, placed in the archive, not only does not decrease, but even increases (due to the fact that the data in the streams is also placed in the archive). This may raise suspicions. A program for monitoring access to the file system - FileMonitor from the same Sysinternals - does not distinguish between access to files or streams. Accordingly, a careful study of the disk access log of a suspicious program (your keylogger) will reveal both the name of the stream where the log is written and the name of the file to which it is attached.
Viruses
In September 2000, the first virus appeared that used alternative file streams to spread. W2k.Stream was the first representative of a new type of virus - stream companion. It looks for .exe files in its directory, and if it finds it, it begins the infection process. An additional stream is attached to the file, into which the virus transfers the contents original file, and then the body of the virus is copied into the main stream of the file. After running the infected file, the virus again tries to infect files in its directory and then launches the program from an additional thread. Indeed, using the CreateProcess function you can start a process from a thread. Moreover, the file with the stream can be safely deleted, but the process will remain. Just a fairy tale for the Trojans! Despite the fact that almost four years have passed since the appearance of W2K.Stream, not all antiviruses are yet capable of detecting malicious code in file streams. Therefore, the emergence of new worms and viruses that use them can pose a serious danger.
Other viruses that use streams
In addition to W2K.Stream, streams have found application in other viruses and worms. The first worm to use file streams was I-Worm.Potok. This little beast attaches multiple threads to the odbc.ini file in Windows directory and stores there scripts for sending itself by mail. Another virus is W2k.Team. Descriptions of these and other similar viruses can be found on the website http://www.viruslist.com/
Working with streams from the console
Creating a file with a stream:
type nul > somefile.txt:Stream
Write to stream:
echo "Something" >> somefile.txt:Stream
Reading from a stream:
more< somefile:Stream
Copying the contents of an existing file to a stream:
type file1.txt >> somefile.txt:Stream
Copying the contents of a stream to a file:
more< somefile.txt:Stream >> file2.txt
Removing threads
There is an opinion that a stream can only be deleted together with the file to which it is attached. This is wrong. If you know the name of the stream, then you can always delete it with the standard DeleteFile function.
Listing 1. Example of creating a thread.
#include
Listing 2. X-Stream: Program showing a list of streams
#include
There is also the following on the topic of file streams:
- NTFS Stream Explorer 2.00 A program for working with NTFS and
In this topic, I will look at four types of metadata that can be attached to a file or directory using the file system NTFS. I will describe for what purposes this or that type of metadata can be used, and I will give an example of its use in some Microsoft technology or third-party software.
We will talk about reparse points, object ids and other types of data that a file may contain in addition to its main content.
Object ID this is 64 bytes that can be attached to a file or directory. Of these, the first 16 bytes allow you to uniquely identify a file within the volume and access it not by name, but by identifier. The remaining 48 bytes can contain arbitrary data.
Object IDs have existed in NTFS since Windows 2000. In the system itself, they are used to track the location of the file that a shortcut (.lnk) refers to. Let's say the file referenced by the shortcut has been moved within the volume. When you launch the shortcut, it will still open. Special Windows service if the file is not found, it will attempt to open the file not by its name, but by a previously created and saved identifier. If the file was not deleted and did not leave the volume, it will open, and the shortcut will again point to the file.
Object identifiers were used in the iSwift technology of Kaspersky Anti-Virus version 7. This is how this technology is described: The technology is developed for file NTFS systems. In this system, each object is assigned an NTFS identifier. This identifier is compared with values in a special iSwift database. If the database values with the NTFS identifier do not match, then the object is checked or rechecked if it has been modified.
However, an overabundance of created identifiers caused problems with disk scanning standard utility chkdsk checks, it's been going on for too long. IN next versions Kaspersky Anti-Virus abandoned using NTFS Object Id.
Reparse Point
On the file system NTFS file or the directory may contain a reparse point, which is translated into Russian as "reprocess point". Special data is added to a file or directory, the file ceases to be regular file and only a special file system filter driver can process it.
Windows contains reparse point types that can be processed by the system itself. For example, reparse points in Windows implement symbolic links (symlinks) and junction points (junction points), as well as mount points for volumes in a directory.
The reparse buffer attached to a file is a buffer with a maximum size of 16 kilobytes. It is characterized by the presence of a tag that tells the system what type the reparse point belongs to. When using a reparse buffer own type You also need to set the GUID in a special field, and it may not be available in Microsoft reparse buffers.
What types of reprocessing points are there? I will list the technologies that use reparse points. These are Single Instance Storage (SIS) and Cluster Shared Volumes in Windows Storage Server 2008 R2, Hierarchical Storage Management, Distributed File System(DFS), Windows Home Server Drive Extender. These are Microsoft technologies, technologies not mentioned here third party companies, using reprocessing points, although there are also such.
Extended Attributes
Extended file attributes. It was about them. It is only worth mentioning here that this technology is practically not used under Windows. Of the software I know of, only Cygwin uses extended attributes to store POSIX permissions. A single file on NTFS can have either extended attributes or a reparse point buffer. Simultaneous installation of both is impossible. The maximum size of all extended attributes in one file is 64 KB.Alternate Data Streams
Additional file streams. Probably everyone already knows about them. I will list the main features of this type of metadata: naming (that is, a file can have several streams, and each has its own name), direct access from the file system (they can be opened using the format “file name, colon, stream name”), unlimited size, the ability to run a process directly from a thread (and the ability to implement it through it).Used in iStream technology of Kaspersky Anti-Virus. They are used in Windows itself, for example, when downloading a file from the Internet, a Zone.Identifier stream is attached to it, containing information about the location from which it was received this file. After running the executable file, the user may see a message “Unable to verify publisher. Do you really want to run this program?.
This gives the user additional protection against the thoughtless launch of programs obtained from the Internet. This is just one use of streams, and they can store a wide variety of data. The mentioned Kaspersky Anti-Virus was stored there checksums each file, but later this technology was also abandoned for some reason.
Anything else?
Is there some more security id, plus standard file attributes that are not directly accessible, even though they are also implemented as file streams. And they, and extended attributes, and reparse and object id - all these are file streams from the point of view of the system. There is no point in directly changing the security identifier, shown in the following picture as::$SECURITY_DESCRIPTOR; let the system deal with changing it. The system itself does not provide direct access to other types of streams. So that's it.View object id contents, reparse points, and work with extended attributes and alternative file streams possible using the program
Most users of modern operating systems of the Windows family have encountered a situation where the help file in CHM (Compiled Help Module) format opens only partially - you can only view the table of contents without the contents of its items:
In addition, when you try to open CHM file, contained in the general network folder, using a UNC (Universal Naming Convention) path such as \\server\h\help.chm, its sections are not displayed. In other words, you can normally view .chm files only if they were not received over the network.
A similar picture is observed when you try to open executable file, which was downloaded from the network. You will see a security warning:
Moreover, the same file, extracted from an archive that was also downloaded from Internet networks, on this computer can open without any problems. In fact, the only difference is that the file being opened was created locally, during the unzipping process, and not downloaded over the network. In other words, Windows has the ability to determine the network origin of a file, and respond to it using certain security settings.
A mechanism for determining the network origin of files.
In the NTFS file system, each file (or directory) is represented as a collection individual elements, called attributes. Elements such as the file name, security settings, and even data are all file attributes. Each attribute is identified by an attribute type code and, optionally, an attribute name. So, for example, the file name is contained in the attribute Filename, content - in the attribute DATA, information about the owner and access rights is in the attribute Security Descriptor etc. The contents of each file ($DATA attribute) is a set streams, in which the data is stored. For each file or directory in NTFS, there is at least one main thread in which the data is actually stored. However, in addition to the main thread, a file or directory can also be associated with alternative (A alternate D ata S stream - ADS), which may also contain some data that is in no way related to the data of the main stream. The main stream of the file has no name and is designated as $DATA:"". Alternative streams must have a name, for example - $DATA:"StreamData"- alternative stream with name StreamDataWhen the functions of writing data to a file are performed, they are placed in the main data stream. When we open, for example, with notepad text file, then we get access specifically to the data of the main thread. The data alternative streams, when using standard access, are not displayed, and indeed, there is not even any sign of their presence. However, alternate stream data associated with a specific file or directory can be accessed using special programs or when using special syntax on the Windows command line.
For example, writing text to the file test.txt with the command echo:
echo Main stream Data > test.txt- write the text “Main stream Data” to a file test.txt, which means writing to the main unnamed stream.
But you can change the command:
echo Alternate stream Data > test.txt:stream1- write the text “Alternate stream Data” to an alternative stream with the name stream1 file test.txt
Now you can open, for example, each of the streams with Notepad:
notepad test.txt- the contents of the main stream will open with the text “Main stream Data”
notepad test.txt:stream1- the contents of the alternative stream will open with the text “Alternate stream Data”
Alternative streams, being invisible to standard tools for working with file system objects, are nevertheless very often used to store additional information about files and other service information. So, for example, when downloading files from the Internet, browsers add an alternative stream called Zone.Identifier, which can be opened with notepad, as in the example above
notepad %USERPROFILE%\Downloads\ChromeSetup.exe:Zone.Identifier- open an alternative stream with the name in notepad Zone.Identifier ChromeSetup.exe You don’t have to specify the path to the file by first executing the command to go to the directory of the current user’s downloaded files (if standard arrangement service user folders):
cd %USERPROFILE%\Downloads- go to the directory of downloaded files.
notepad ChromeSetup.exe:Zone.Identifier- open an alternative stream with the name Zone.Identifier for the browser installation file Google Chrome With name ChromeSetup.exe in the current directory.
As you can see, the contents of the alternative stream contain the lines:
- section sign with a description of the data transmission area
ZoneId=3- zone identifier.
This information makes it possible to determine the origin of the file by identifier number ZoneId:
0
- local computer(Local).
1
- local the local network(Intranet)
2
- Trusted Sites
3
- Internet
4
- dangerous sites (Restricted Sites)
This definition of zones, for example, corresponds to the security settings of Internet Explorer:
IN in this case, you can determine that the file ChromeSetup.exe was obtained from the Internet (zone ID = 3). When running such a file, a security warning will be issued about unreliable source. Application security works in a similar way. Microsoft Office, when they warn about the dangers of opening files that have been downloaded from the Internet. For the same reason, the contents of help files in the CHM format do not open - the contents of the alternative stream allow them to be classified as dangerous, regardless of the actual or non-existent danger.
Try changing the same notepad, the ZoneId value to 0 , which will correspond to the local origin of the file, and the security warning will disappear, as will problems with opening office documents or help topics in .chm files.
A similar behavior of security systems will occur in cases where the contents of the alternative stream are deleted (make it empty), or the alternative stream itself is deleted altogether.
Starting in Windows 7, you can use the command to get a list of alternative file streams DIR with parameter /R:
dir /r %UserpRofile%\Downloads- display a list of files and alternative streams in a directory Downloads current user.
To work with alternative streams in any version of Windows OS, you can use the utility streams.exe from the composition software package Microsoft Sysinternals Suite. The package contains many small programs for diagnostics, optimization and administration, including a utility that allows you to compensate for shortcomings in working with alternative streams.
Command line format:
streams.exe [-s] [-d]file or directory
Command line options:
-s- process subdirectories.
-d- delete alternative streams.
-nobanner- do not display the start banner and copyright information.
Examples of using:
streams.exe /?- display help on using the program.
streams myfile.txt- display information about file streams myfile.txt
streams –d myfile.txt- remove alternative file streams myfile.txt
streams -d -s D:\Downloads\*.*- delete alternate streams of all files and subdirectories in a directory D:\Downloads\
IN operating systems Windows 8 and later, PowerShell also allows you to work with alternate threads:
Get-Item -Path -Path C:\FirefoxSetup.exe -Stream *- display information about threads in the file C:\FirefoxSetup.exe.
Get-Content -Path C:\FirefoxSetup.exe -Stream Zone.Identifier- display the contents of an alternative stream Zone.Identifier file C:\FirefoxSetup.exe
Remove-Item -Path C:\FirefoxSetup.exe -Stream *- remove all alternative streams associated with the file C:\FirefoxSetup.exe
Remove-Item -Path C:\FirefoxSetup.exe -Stream Zone.Identifier- delete alternative stream Stream Zone.Identifier associated with the file C:\FirefoxSetup.exe.
Security zone information is widely used in group policies, and in particular, the Windows Attachment Manager, which performs protection functions against malware that may be contained in mail attachments or files downloaded from the Internet. The Microsoft website contains a detailed article on how to configure the Attachment Manager and resolve problems associated with it:
Description of the operation of the attachment manager, which is included in the Microsoft Windows system.
In conclusion, I will add that alternative streams are a property of the NTFS file system, and, for example, are not supported in FAT32. Accordingly, when copying files from NTFS to any other file system, alternative streams are discarded.
Alternate Data Stream (AltDS) support was added to NTFS for compatibility with the Macintosh file system HFS, which used a resource stream to store icons and other file information. The use of AltDS is hidden from the user and is not accessible through normal means. Explorer and other applications work with the standard stream and cannot read data from the alternative stream. With AltDS you can easily hide data that cannot be detected by standard system checks. This article will give basic information about the working and definition of AltDS.Creation of AltDS
Creating an AltDS is very easy. To do this, we will use the command line. First, let's create a base file to which we will attach our streams.C:\>echo Just a plan text file>sample.txtC:\>type sample.txt
Just a plan text file
Next, we'll use a colon as an operator to indicate that we'll be using AltDS:
C:\\>echo You can"t see me>sample.txt:secret.txt
You can use the following commands to view the content:
C:\more< sample.txt:secret.txt
or
C:\notepad sample.txt:secret.txt
If everything works well, you will see the text: You can't see me, but when opened from Explorer, this text will not be visible. You can also attach AltDS not only to a file, but also to a folder. To do this, create a folder and attach some kind of some text:
C:\>md stuff
C:\>cd stuff
C:\stuff>echo Hide stuff in stuff>:hide.txt
C:\stuff>dir
Volume in drive C has no label.
Volume Serial Number is 40CC-B506Directory of C:\stuff
09/28/2004 10:19 AM.
09/28/2004 10:19 AM…
0 File(s) 0 bytes2 Dir(s) 12,253,208,576 bytes free
C:\stuff>notepad:hide.txt
Now you know how to view and edit an attached AltDS using Notepad, as well as how to attach it to files and folders.
Hiding and launching applications
Hiding applications using AltDS is as easy as hiding test files. First, let's create the base file again:Next, let's place our application in a stream, for example I used notepad.exe:
C:\WINDOWS>type notepad.exe>test.txt:note.exe
Now let’s make sure that our file contains the same text:
C:\WINDOWS>type test.txt
Test
And now the fun part, let’s launch our hidden application:
C:\WINDOWS>start .\test.txt:note.exe
C:\WINDOWS>
Since this article is not a complete translation of the article taken, it is formatted as a simple topic. Additional techniques can be found at the link provided.
UPD:
Utilities for working with AltDS (list taken from the article linked above):
LADS - List Alternate Data Streams by Frank Heyne
www.heysoft.de/Frames/f_sw_la_en.htm
Streams.exe from SysInternals.
CIOs spend a lot of time and resources on systems-related projects analytical processing sales information and other standard business data. At the same time, dashboards are created for managers to display company performance indicators and help them make forecasts for the future. Such systems bring significant business benefits, but in fact, the opportunities they open up are only a small part of what can be done with the data available to the organization, says Krishna Nathan, CIO of S&P Global (formerly McGraw Hill Financial), which deals with credit management. ratings, as well as providing consulting and analytical services for the stock market. Under Nathan's leadership, a new enterprise-wide data processing system was designed and implemented, implementing a strategy aimed at accelerating business growth and creating new offerings for customers.
Some companies are starting to collect additional data - they call it alternative, non-traditional or orthogonal. While this is still new, CIOs should become familiar with the technology today. After all, very soon alternative data will become a mandatory tool for many companies.
However, do not rush to hire yet another expensive specialist. Let's figure out what we're actually talking about.
What is “alternative data”
Nathan defines alternative data as follows: it is data that comes from non-traditional sources and can be analyzed to extract useful information in addition to what you normally receive.
Let's say you have a retail chain and you intend to open new shop in another city. Typically, such a decision is based on the performance of your stores in a particular city and other cities.
An alternative source of data here could be photographs of supermarket parking lots taken over several months - parking occupancy levels can be correlated with sales volumes. As well as information about pedestrian traffic in the area where the store is planned to open. By combining the information you receive, you can learn something new that will help you in your business.
S&P Global also provides analytics services for commodity exchanges, and the CIO has to constantly think about how to offer customers additional insights using alternative data sources, how to combine various information to give customers information they couldn't get anywhere else.
Let's say S&P Global has information that an oil refinery in Rotterdam can produce 100 thousand barrels of petroleum products per day. But due to supply shortages, approximately 70 thousand barrels are processed, that is, free capacity for another 30 thousand is available. What happens after an oil tanker with 30 thousand barrels enters the port? “If the plant's available capacity report is from a week ago, we won't know that the oil has just been unloaded,” explains Nathan. – That is, traditional data is outdated. This is where a source of alternative data such as satellite imagery comes in handy. If we analyze satellite imagery along with other sources, we will get a more accurate picture of reserves and production in almost real time.”
Alternative Data and the CIO
Even if you don't have ready-made application scenarios, get acquainted with new technologies. Plan systems that allow you to combine multiple data sources for analysis. Learn to manage the data delivery chain, protect it, and take into account usage rights. And hire the necessary staff - you need experienced data scientists who can analyze data and extract useful information.
For quick launch project in the field of alternative data, you can use a ready-made solution. This is what S&P Global did when Platts, a subsidiary of the company, acquired cFlow, an interpretation toolkit satellite imagery. CFlow offers tools visual representation data that allows you to monitor changes in trade flows along the routes of ships, provides information on the volume and nature of tanker cargo.
Convince company management that the time has come to invest in alternative data - buying existing solutions or creating your own. Some of your alternative data projects will work, but many will fail. Well, if alternative data brings truly valuable information, use it to receive funds for new projects.
– Martha Heller What is ‘alternative data’ and how can you use it? CIO. JAN 3, 2017