Visibly-invisibly

Blog reader Victor was unable to launch the downloaded from the Internet PowerShell script. Carefully reading my instructions avoided the problem, but the root of it was not PowerShell's strict security policies.

Victor downloaded an archive from the TechNet gallery with the PSWindowsUpdate.zip script for managing Windows Update which I was talking about. However, the unpacked script refused to work. When I suggested to the reader that the first point of my instructions talked about the need to unlock the archive, everything went like clockwork.

Victor asked to explain why the system blocked the script, and how it knew that the archive was downloaded from another computer.

To be honest, today's topic is not new, but I decided to cover it on my blog for several reasons:

• Many articles were written back in Windows times XP or Windows 7 and do not take into account the built-in capabilities of newer Microsoft operating systems.
• One of the articles planned for the near future touches on this topic, and it is easier for me to refer to material for the relevance and correctness of which I myself am responsible.
• The blog has a large audience, and for many readers this topic will still be new :)

Today on the program

NTFS data streams

Windows gets information about the file's source from the alternative data stream (ADS) of the NTFS file system. In the file properties, she modestly writes that it is from another computer, but in reality she knows a little more, as you will see later.

From an NTFS point of view, a file is a collection of . The contents of the file is a data attribute named $DATA. For example, a text file with the line “Hello, World!” has the data attribute “Hello, World!”

In NTFS, the $DATA attribute is a data stream and is called primary or unnamed because... it has no name. Formally, it looks like this:

$DATA:""

• $DATA- Name attribute
• : - delimiter
• "" - Name flow(V in this case the name is missing - there is nothing between the quotes)

Interesting Features of Alternative Data Streams

In the context of the examples above, I want to make a few interesting points.

Invisible changes

Having created a text file with the first command, you can open it in text editor and make sure that all further manipulations do not affect the contents of the file in any way.

It gets interesting when the file is opened, say, in Notepad++. This editor can warn you about file changes. And it will do this when you write an alternative stream to the file, but the content will remain the same!

Record and view ADS from CMD

ADS can be created and displayed from the command line. The following commands write hidden text to a second ADS named MyStream2 and then display it.

Echo Hidden Text > C:\temp\test.txt:MyStream2 more< C:\temp\test.txt:MyStream2

Viewing ADS in text editors

The same Notepad++ will show you the contents of ADS if you specify the name of the stream in command line

"C:\Program Files (x86)\Notepad++\notepad++.exe" C:\temp\test.txt:MyStream1

Result:

With notepad, this trick will work only if there is a .txt. The commands below add a third ADS and open it in Notepad.

Echo Hidden Text > C:\temp\test.txt:MyStream3.txt notepad C:\temp\test.txt:MyStream3.txt

Result:

Blocking downloaded files

Let's get back to the question a reader asked me. Whether a file will be blocked depends primarily on the program in which it was downloaded, and secondly on the OS settings. So, all modern browsers support blocking, and it is included in Windows.

Remember that when an archive is locked, all unpacked files will be locked “inheritedly”. Also remember that ADS is a feature of NTFS, i.e. When saving or unpacking an archive on FAT32, no blocking occurs.

View information about the source of a blocked file

In PowerShell, go to the folder with the downloaded file and see information about all the threads.

Get-Item .\PSWindowsUpdate.zip -Stream * FileName: C:\Users\Vadim\Downloads\PSWindowsUpdate.zip Stream Length ------ ------ :$DATA 45730 Zone.Identifier 26

As you already know, $Data is the contents of the file, but ADS also appears in the list Zone.Identifier. This is a clear hint that the file was received from some zone. Do you know where this picture is from?

To find out the zone, you need to read the contents of the ADS.

Get-Content .\PSWindowsUpdate.zip -Stream Zone.Identifier ZoneId=3

Obviously, it is aimed at batch unlocking (for example, when the archive is already unpacked). The command below will unlock in the Downloads folder all files containing in the name PS:

Dir C:\Downloads\*PS* | Unblock-File

Of course, there are all sorts of utilities with graphical interface, even those who know how to integrate into context menu. But, in my opinion, PowerShell or, at worst, streams is quite enough.

How to prevent files from being blocked

The blocking is controlled by the group policy Do not store information about the zone of origin of attached files. As the name suggests, blocking is standard Windows behavior, and the policy allows you to change it.

However, it is not obvious from the name that the policy applies not only to mail attachments, but also files downloaded from the Internet. Read more about the attachment manager in KB883260.

There is no group policy editor in home editions, but no one has canceled the registry: SaveZoneInformation.zip.

Other examples of practical application of ADS

The scope of ADS is not limited to adding a downloaded file zone, nor is it necessary to store only text in ADS. Any program can use this NTFS feature to store any kind of data, so I'll just give a couple of examples from different areas.

File Classification Infrastructure

about the author

Interesting material, thank you. I learned something new about PowerShell, which is still a little familiar to me :)

I often use WhatsApp to communicate with my family - so far there have been the fewest problems with this service, even my parents have gotten used to it. Kontaktik is also mainly for family, although the exchange of messages there is mainly around published albums with photos and videos. Some relatives remain faithful to Viber - it didn’t work out for me, I just keep it for them, without giving up trying to drag them to WhatsApp.

For work, mainly Slack, when something is urgent - WhatsApp, very urgent - SMS. VKontakte for communication about working with the outside world.

I use Skype only for video calls, mainly with my family. I would gladly replace it with WhatsApp if there were video calls.

urix

Viber now has video calls, and even video calls for the desktop version. So maybe Viber will be the next Skype... in in a good way

Andrey Kuznetsov

Interesting material, thank you. I knew about the existence of threads, but I didn’t know that it was so easy to work with them through PowerShell.
As for IM: I only have complaints about Skype regarding the launch time Windows Phone. There is no such problem on iPad and Windows. I use it for voice communication, when for some reason it is inconvenient to use GSM.
And correspondence via Whatsapp. Having it only on the phone is rather a plus, from a privacy point of view.

• Andrey Kuznetsov: And correspondence via Whatsapp. Having it only on the phone is rather a plus, from a privacy point of view.

  Andrey, explain what is the plus here?

Pavlovsky Roman

1. I use most often: Skype and Hangouts - for work on a PC, for other correspondence on VKontakte from any device, since work clients usually use Skype, and friends and acquaintances on Social Networks.

2. I would ideally like to use: Jabber - for correspondence and calls from any device. As for me, the client can be installed on any device and correspond with each other wherever the user is, even on a weak Internet connection + in addition, you can deploy your own jabber server and store all correspondence on the server, so that later you can quickly find the necessary correspondence, if the client does not know how to store history, and plugins for calls via Jabber can be found (for example, through the same SIP Asterisk 1.8+)

Andrey Bayatakov

Most often I use WhatsApp (mainly for work), for calls (audio/video/international calls) Skype. Although desktop Skype is terribly infuriating (I have a transformer and at home I use it mainly as a tablet)… Viber has not caught on. To make calls via WhatsApp you just need to have nerves of steel. You say something to your interlocutor and wait a minute or two for him to hear you (50Mbit connection)…
If there was an opportunity, I would completely switch to Skype. On Windows 10 Mobile, after a recent update, messages from Skype come directly to the built-in Messages application (like SMS), which is very convenient.

Maxim

1. Reluctantly, I use ICQ (for retrograde customers) and Slack (for more modern ones).
2. I would like to use Jabber - for the same reasons as Roman Pavlovsky above.

Vladimir Kiryushin

Hello Vadim!
Before this article, I read your article about how to read a check report for everything system disk with the chkdsk command. Great article! Thanks to her, today after checking the system disk with the chkdsk command, I received a text file of the report. And this article also clarifies a lot of things in PowerShell program. Some things are incomprehensible to me as a pensioner, but I try not to panic and read diligently to the end. Thank you for the study you are doing with us! All the best to you!

Lecron

What browsers and downloaders create this stream?

What other options are there for the user to use threads? And in particular, a script writer user? Because, although I knew about them for a long time, I never used them. At real work with a computer you simply don’t remember about them, and because of this, you may end up using crutches instead of a convenient tool, and without this work, from memory, you can’t come up with anything.
I only thought of one option. Comment on the file if there is no opportunity or desire to write long text to the file name. But this requires support from the file manager, who previously, and even now, writes them to descript.ion or files.bbs.

Speed ​​Guru

Another garbage technology like USN magazine. How much use will you get from ZoneIdentifier or from a virus attached to a file or folder? Of course not. Moreover, this is cluttering the system with unnecessary “sub-files” that are in no way needed by a normal user. Each extra read in the MFT directory and other operations associated with the maintenance and maintenance of alternative threads are extra spent processor cycles, random access memory, and most importantly, an extra load on the hard drive.
You can tell me that this technology is very necessary for the system. But this is nonsense - the system would work perfectly without threads. But no one asks the user - they sold it (like a USN magazine) and did not give the opportunity to completely disable the maintenance of these flows. But as users, I don’t need them at all, I think like you…
All we can do is “streams -s -d %systemdrive%”. But this also does not make it possible to delete threads on the system partition.

Alexiz Kadev

Named streams are a great thing, and they existed, as far as I remember, from the first release of NTFS. It’s quite convenient to store, for example, document versions in named streams, which, if I’m not mistaken, a number of applications have done. But there remains an ambush with copying to another file system- named streams are simply cut off.

It’s a pity that it was impossible to select several messengers in the poll: I use several, since some of my contacts prefer certain ones. So, I use WhatsUp, ICQ (though, of course, not a native client), Skype, SkypeforBusiness (quiet horror, not a client, however, when it was called Lync it was even worse) and Viber (here there is more spam than in others at least once at 5).
And ideally, use just one, like Miranda with plugins, since finding, if necessary, who said/wrote something when in this whole bunch is simply unrealistic. But alas, a number of manufacturers close their protocols and protect them like Kashchei protects his needle.

Vladimir Kokarev

VSh

Vadim Sterkin: Roman, I did not include Jabber in the survey. I decided that few people use it and there are no prospects.

In vain
For example, I use OpenFire (freeware xmpp) as an office communicator on several domains.

Therefore, my main one is XMPP (Pidgin.exe, Spark.exe), but 99.8% of these messages are intradomain.
Skype - for external IM
WhatsApp and Viber are for “random connections”, the last n months have been nothing but SPAM, I’m thinking – should I delete it?

Artem

For some reason everything is on Viber. And the quality of communication is quite satisfactory. Otherwise there would be a telegram. It's empty there.

hazet

1. Skype (on PC) and Viber (on Mobile). The reasons are basically the same as for most - the number of available contacts and, naturally, the reluctance of these same contacts to switch to another messenger.
2.uTox. Miniature, nothing superfluous, client for Win, Linux, Mac and Android. Positioned as protected.
P.S. I'll start dragging my contacts onto it more tightly :-)

Evgeniy Karelov

Thank you for your work!

Regarding the survey, on a PC for correspondence I use QIP 2012, to which I am connected ICQ contacts, VKontakte and others. Personally, it’s convenient for me to use one program to communicate over several protocols. And the ability to view social media feeds from one place is very pleasing. Ideally, the only thing missing is support for Skype, which I use for voice communication, but it obviously won't appear.
Although this program looks “abandoned”, because there have been no updates for a long time, it performs its assigned functions perfectly.

strafer

An interesting mixture of the topic of the post about data flows and the IM survey.

According to the survey: Jabber/Jabber, which you shouldn’t have included in the list, although there is WhatsApp based on XMPP, and even Asechka, which is heading towards success.

Jabber basically solves everything mentioned problems due to the openness of the protocol, the presence of clients for many platforms and the availability of servers that can be set up independently. But chewing cacti is more traditional, yes.

• The list includes clients, not protocols.
  ICQ... well, I didn’t put emoticons there, because it should be clear.
  Jabber definitely doesn't solve one problem - no one is there.

  • strafer

    Vadim Sterkin: Clients are listed, not protocols.

    Due to the fact that the protocol and source codes official client are closed, a natural identity is established between the only client and the protocol.

    Vadim Sterkin: ICQ... well, I didn’t put emoticons there, because it should be clear.

    It is not enough for the rotten mail girl that the asechka dies a natural death - they also make additional efforts to make it die faster.

    Vadim Sterkin: Jabber definitely doesn't solve one problem - there's no one there.

    Nevertheless, you yourself wrote for Telegram

    looks great, but it's empty (which can be fixed)

    Jabber had every chance of becoming what the e-mail ecosystem is today (complete openness of the protocol, the ability to set up your servers for anyone and ensure interaction between servers, etc.), but corporations do not need this, which is clearly seen in the example of the departure from him Google or proprietary WhatsApp.

    • For Telegram - fixable, for Jabber - very unlikely. Therefore, the first one is on the list, but the second one is not.

      • strafer

        Of course, Telegram is stylish, fashionable, youthful, but Jabber is not used by anyone cool like Pasha Durov. What are the prospects here?

        Hm... come out of your tank of “the whole world is against free software” conspiracy theories. All much easier

        

        If it’s not clear, this is what a person’s first experience of interacting with the officially recommended Jabber client on the most common mobile platform looks like.

        strafer

      • I didn’t understand a little where in my comment about the conspiracy.

        Yes, everywhere :) You are trying to attribute the failures of jabber to unfashionability and lack of youth, while its clients from the first screen are not adapted to modern reality.

        What should I see in the screenshot?

        Prompt to enter a phone number ~~~O~

      strafer

      strafer: You are trying to attribute the failures of jabber to being unfashionable and not youthful

      Well, if that's the case.

      strafer: while its clients from the first screen are not adapted to modern reality.

      Those. to the current fashion, such as disclosing your phone number to everyone. Because I don’t understand why it should be introduced if it is not needed for the operation of the system, as for me it is absolutely wonderful that it is not asked here.

      Actually, I abandoned the account, despite the few remaining contacts there, precisely for this reason - Meirushechka, in the form of an ultimatum, demanded to link the phone number to the account, as a result of which she was sent to known coordinates.

      Yes, you don’t understand, even after explanations with pictures... This is not fashion, this the only way make registration as easy as possible with mobile devices, which form the basis of the audience of modern messengers and the only source for its growth.

      strafer

      The screenshot shows a request for a name, password and optional nickname. Where should we simplify more? Or, apart from the students of special schools, there are no more reserves left for audience growth, and there needs to be one button “do it for the sake of it”?
      Why is there a phone number at all and what should the messenger do with the phone number?

In this topic, I will look at four types of metadata that can be attached to a file or directory using the file system NTFS. I will describe for what purposes this or that type of metadata can be used, and I will give an example of its use in some Microsoft technology or third-party software.

We will talk about reparse points, object ids and other types of data that a file may contain in addition to its main content.

Object ID this is 64 bytes that can be attached to a file or directory. Of these, the first 16 bytes allow you to uniquely identify a file within the volume and access it not by name, but by identifier. The remaining 48 bytes can contain arbitrary data.

Object IDs have existed in NTFS since Windows 2000. In the system itself, they are used to track the location of the file that a shortcut (.lnk) refers to. Let's say the file referenced by the shortcut has been moved within the volume. When you launch the shortcut, it will still open. Special Windows service if the file is not found, it will attempt to open the file not by its name, but by a previously created and saved identifier. If the file was not deleted and did not leave the volume, it will open, and the shortcut will again point to the file.

Object identifiers were used in the iSwift technology of Kaspersky Anti-Virus version 7. This is how this technology is described: The technology was developed for the NTFS file system. In this system, each object is assigned an NTFS identifier. This identifier is compared with values ​​in a special iSwift database. If the database values ​​with the NTFS identifier do not match, then the object is checked or rechecked if it has been modified.

However, an overabundance of created identifiers caused problems with disk scanning standard utility chkdsk checks, it's been going on for too long. IN next versions Kaspersky Anti-Virus abandoned using NTFS Object Id.

Reparse Point

On the file system NTFS file or the directory may contain a reparse point, which is translated into Russian as "reprocess point". Special data is added to a file or directory, the file ceases to be regular file and only a special file system filter driver can process it.

Windows contains reparse point types that can be processed by the system itself. For example, reparse points in Windows implement symbolic links (symlinks) and junction points (junction points), as well as mount points for volumes in a directory.
The reparse buffer attached to a file is a buffer with a maximum size of 16 kilobytes. It is characterized by the presence of a tag that tells the system what type the reparse point belongs to. When using a reparse buffer own type You also need to set the GUID in a special field, and it may not be available in Microsoft reparse buffers.

What types of reprocessing points are there? I will list the technologies that use reparse points. These are Single Instance Storage (SIS) and Cluster Shared Volumes in Windows Storage Server 2008 R2, Hierarchical Storage Management, Distributed File System(DFS), Windows Home Server Drive Extender. These are Microsoft technologies, technologies not mentioned here third party companies, using reprocessing points, although there are also such.

Extended Attributes

Extended file attributes. It was about them. It is only worth mentioning here that this technology is practically not used under Windows. From what I know software only Cygwin uses extended attributes to store POSIX permissions. A single file on NTFS can have either extended attributes or a reparse point buffer. Simultaneous installation of both is impossible. Maximum size of all extended attributes for one file is 64 KB.

Alternate Data Streams

Additional file streams. Probably everyone already knows about them. I will list the main features of this type of metadata: naming (that is, a file can have several streams, and each has its own name), direct access from the file system (they can be opened using the format “file name, colon, stream name”), unlimited size, the ability to run a process directly from a thread (and the ability to implement it through it).

Used in iStream technology of Kaspersky Anti-Virus. They are used in Windows itself, for example, when downloading a file from the Internet, a Zone.Identifier stream is attached to it, containing information about the location from which it was received this file. After running the executable file, the user may see a message “Unable to verify publisher. Do you really want to run this program?.

This gives the user additional protection against the thoughtless launch of programs obtained from the Internet. This is just one use of streams, and they can store a wide variety of data. The mentioned Kaspersky Anti-Virus stored checksums of each file there, but later this technology was also abandoned for some reason.

Anything else?

Is there some more security id, plus standard file attributes to which there are no direct access, despite the fact that they are also implemented as file streams. And they, and extended attributes, and reparse and object id - all these are file streams from the point of view of the system. There is no point in directly changing the security identifier, shown in the following picture as::$SECURITY_DESCRIPTOR; let the system deal with changing it. The system itself does not provide direct access to other types of streams. So that's it.

Viewing the contents of object id, reparse points, as well as working with extended attributes and alternative file streams is possible using the program

Alternative data streams in NTFS

The NTFS file system has many interesting features, one of which is the presence of alternative data streams (Alternate Data Stream, ADS). Their essence is that each file in NTFS is a set of streams in which data is stored. By default, all data is in the main stream, but if necessary, additional, alternative data streams can be added to the file.

Note. Alternative data streams in NTFS appeared a long time ago, back in Windows NT. They were created for compatibility with the HFS file system, then used on MacOS. HFS stored file data in a special resource stream.

Files in NTFS are divided into attributes, one of which is the $DATA, or data attribute. Streams are additional properties of the $DATA attribute. By default there is one, main thread $DATA:″″. As you can see, it has no name, so it’s called unnamed. You can also create additional named streams if you wish, for example. $DATA:″Stream1″. Each file in NTFS can have several data streams containing different, unrelated data.

All data written to the file ends up in the main data stream by default. When we open a file, we see exactly the main stream, while alternative streams are hidden from the user and are not displayed using normal means. They cannot be seen in standard ways, although some programs can read the data hidden in them. You can also use the command line to work with streams.

For example, let's open the console and use the echo command to create a text file streams.txt and write the text into it:

echo This is main stream>streams.txt

A next command write the text to the alternative stream stream1:

echo This is alternate stream>streams.txt:stream1

If we now open the streams.txt file in any text editor, we will see only the first entry, the text “This is alternate stream” will remain hidden. You can read the information hidden in stream1 with the command:

more

Alternative streams can be added not only to individual files, but also to directories. For example, let's add an alternative stream stream2 containing the text “Hide stream in Streams” to the current Streams directory:

echo Hide stream in Streams>:stream2

And output stream2 with the following command:

more<:stream2

Alternate stream content can be opened in more than just the console. For example, Notepad can also access data hidden in streams if you specify the name of an alternative stream in the file name, separated by a colon. Let's repeat the previous example, slightly changing the stream name to stream1.txt:

echo This is alternate stream>streams.txt:stream1.txt

And open an alternative stream in notepad with the command:

notepad streams.txt:stream1.txt

Note. Standard Notepad requires a txt extension in the stream name, otherwise it will not be able to open it. More advanced editors, such as Notepad++, can show the contents of an alternative stream, regardless of its name.

The presence of alternative streams in a file is not displayed in any way in Explorer and others. file managers. In order to find them, the easiest way is to use the command dir/R(beginning with Windows Vista), which shows all data streams, including alternative ones.

You might think that alternative streams are limited to text data. This is not true at all, and absolutely any information can be stored in alternative streams. For example, let's create a picture.txt file and add the pic1.jpg stream to it, into which we will place the image of the same name:

echo Picture>picture.txt
type pic1.jpg>picture.jpg:pic1.jpg

Thus, externally we have a regular text file, but to open an image from an alternative stream in graphic editor Paint we will use the command:

mspaint picture.txt:pic1.jpg

In a similar way, you can add any data to any type of file - add images to text files, add text information etc. Interestingly, alternative content does not increase the apparent file size, for example adding to 1kB text file 30GB HD video, Explorer will still show the file size as 1kB.

You can also hide in alternative streams executable files. For example, take the test.txt file and add the Notepad application (notepad.exe) to the alternative note.exe stream:

type notepad.exe>test.txt:note.exe

And to launch a hidden notepad we will use the command:

start .\test.txt:note.exe

By the way, some malicious programs take advantage of this opportunity by adding executable code to alternative NTFS streams.

Streams utility

There are several options for working with alternative streams. third party utilities, for example the Streams console utility from Sysinternals. It can detect the presence of alternative threads and remove them. The utility does not require installation, just unpack it and run it. For example, let's check the presence of streams in the Streams folder with the command:

Streams.exe -s C:\Streams

And remove alternative streams from the streams.txt file:

Streams.exe -d C:\Streams\streams.txt

PowerShell

PowerShell can also work with alternative streams - create, detect, display their contents, and even delete them. For example, let's create a text file:

New-Item -Type file -Path C:\Streams\stream.txt

Let's add an entry to the main thread:

Set-Content -Path C:\Streams\stream.txt -Value ″Main stream″

And to an alternative stream named Second:

Set-Content -Path C:\Streams\stream.txt -Value ″Second stream″ -Stream Second

Then we will output the contents of the main

Get-Content -Path C:\Streams\stream.txt

and alternative streams:

Get-Content -Path C:\Streams\stream.txt -Stream Second

In order to detect the presence of alternative streams, you can use the command:

Get-Item -Path C:\Streams\stream.txt -Stream *

And you can remove unnecessary threads with the command:

Remove-Item -Path C:\Streams\streams.txt -Stream *

Usage

Alternative streams are used both by Windows itself and by some programs. Eg, Internet Explorer divides the network into 4 security zones and, when downloading files, adds tags to them that contain information about the zone from which they were downloaded.

These tags are stored in an alternative stream and represent a number from 0 to 4:

Internet (3)
Local network (1)
Trusted sites (2)
Dangerous sites (4)
Local computer (0)

To make sure of this, let's go to the downloads folder, take a file downloaded from the Internet and check it for the presence of alternative streams. As you can see, it contains a thread named Zone.Identifier, which contains the string ZoneID=3.

This means that the file belongs to an untrusted Internet zone, and you need to be careful when opening it. Some programs, such as Word, read this data when you open the file and issue a warning accordingly.

Also, the File Classification Infrastructure (FCI) is based on the use of alternative streams. From third party programs alternative streams are used by some antivirus programs, in particular, Kaspersky Anti-Virus stores in them the checksum obtained as a result of the scan.

However, the use of alternative streams is not limited to this; you yourself can come up with any use for them. For example, with their help you can hide from prying eyes personal information. Files containing alternative streams can be freely copied or transferred from disk to disk, all streams will be copied along with the file.

And yet, when using alternative streams, you must remember that they are strictly tied to the NTFS file system. In order to use them, the files must be located on NTFS disks, so you can only work with them from Windows. If you move the file to any other file system, then all streams except the main one will be lost. Also, alternative streams are cut off when transferring files via FTP or when sending them as an email attachment.

CIOs spend a lot of time and resources on systems-related projects analytical processing sales information and other standard business data. At the same time, dashboards are created for managers to display company performance indicators and help them make forecasts for the future. Such systems bring significant business benefits, but in fact, the opportunities they open up are only a small part of what can be done with the data available to the organization, says Krishna Nathan, CIO of S&P Global (formerly McGraw Hill Financial), which deals with credit management. ratings, as well as providing consulting and analytical services for the stock market. Under Nathan's leadership, a new enterprise-wide data processing system was designed and implemented, implementing a strategy aimed at accelerating business growth and creating new offerings for customers.

Some companies are starting to collect additional data - they call it alternative, non-traditional or orthogonal. While this is still new, CIOs should become familiar with the technology today. After all, very soon alternative data will become a mandatory tool for many companies.

However, do not rush to hire yet another expensive specialist. Let's figure out what we're actually talking about.

What is “alternative data”

Nathan defines alternative data as follows: it is data that comes from non-traditional sources and can be analyzed to extract useful information in addition to what you normally receive.

Let's say you have a retail chain and you intend to open new shop in another city. Usually similar solution based on the performance of your stores in a specific city and other cities.

An alternative source of data here could be photographs of supermarket parking lots taken over several months - parking occupancy levels can be correlated with sales volumes. As well as information about pedestrian traffic in the area where the store is planned to open. By combining the information you receive, you can learn something new that will help you in your business.

S&P Global also provides analytics services to commodity exchanges, and the CIO has to constantly think about how to offer customers alternative data sources. additional information how to combine various information to give customers information they couldn't get anywhere else.

Let's say S&P Global has information that an oil refinery in Rotterdam can produce 100 thousand barrels of petroleum products per day. But due to supply shortages, approximately 70 thousand barrels are processed, that is, free capacity for another 30 thousand is available. What happens after an oil tanker with 30 thousand barrels enters the port? “If the plant's available capacity report is from a week ago, we won't know that the oil has just been unloaded,” explains Nathan. – That is, traditional data is outdated. This is where a source of alternative data such as satellite imagery comes in handy. If we analyze satellite imagery along with other sources, we will get a more accurate picture of reserves and production in almost real time.”

Alternative Data and the CIO

Even if you don't have ready-made application scenarios, get acquainted with new technologies. Plan systems that allow you to combine multiple data sources for analysis. Learn to manage the data delivery chain, protect it, and take into account usage rights. And hire the necessary staff - you need experienced data scientists who can analyze data and extract useful information.

For quick launch project in the field of alternative data, you can use a ready-made solution. This is what S&P Global did when Platts, a subsidiary of the company, acquired cFlow, an interpretation toolkit satellite imagery. CFlow offers tools visual representation data that allows you to monitor changes in trade flows along the routes of ships, provides information on the volume and nature of tanker cargo.

Convince company management that the time has come to invest in alternative data - buying existing solutions or creating your own. Some of your alternative data projects will work, but many will fail. Well, if alternative data brings truly valuable information, use it to receive funds for new projects.

– Martha Heller What is ‘alternative data’ and how can you use it? CIO. JAN 3, 2017

DIR /B C:\WINDOWS\System32\*.SCR

DIR /B C:\WINDOWS\System32\*.* |FIND /i ".SCR"

Describe in detail the purpose of the parameters of each command (remember that for each command you can call help with the /? key). Please note that the same keys can have different action for different teams.

4.1.8. NTFS* file streams

The NTFS file system supports file streams - alternative data streams. In fact, file streams are a combination of several files into one group with one common file name (each stream has its own additional name). Within a group there is a main data stream, which most programs work with as a file, and additional named streams that are not displayed by normal means. At file operations copying, moving, deleting, etc., in NTFS the operation is performed on the entire group. When using some archivers and copying files containing alternative streams to a FAT partition, these streams may be lost. Technically, alternative streams are used to supplement a file with information without changing the contents of the main stream and without creating additional files, which may be lost.

Alternative streams are used by antiviruses to save information about a file (“fingerprint”, checksum) to detect changes in the file over time. Clients of the exchange system Direct files Connect (DC++) can store hashing results (calculating checksums) For large files, which are used in the case of moving a file during re-hashing, which significantly speeds up updating the list.

In the future, library programs, film libraries and audio libraries can use alternative streams to store, together with documents, streams of covers, audio tracks, descriptions, and on various languages. Alternate streams allow “secret” data to be attached, which is a potential danger.

You can view information about streams using the STREAMS 25 command, NTFS program Stream Explorer26, using file manager extensions27, in Windows 7, the dir /r command displays a list of all streams for the specified objects (you can also use additional switches with the dir command).

When saving files from the Internet, by default a Zone.Identifier 28 stream is added to the file in NTFS, which has an ini file format and usually contains the text:

The ZoneId parameter with a number means the zone from which the file arrived on the computer; the zone number is taken from the security zone settings ( Control Panel/Internet Options(Network and Internet / Browser Properties -

Zera )/Security tab). The following values ​​are allowed29: 0 – local computer

1 – intranet (local network, domain)

2 – trusted source

3 – Internet

4 – untrusted source

If the value is 3, the system will issue a warning “ Cannot check

rip the publisher. Do you really want to run this program?"

At the bottom of the message there is a checkbox " Always ask when opening this file", removing which removes the Zone.Identifier stream. If ZoneId contains a value of 4, a warning will appear " These files cannot be opened. Internet security settings prevented you from opening

25 Streams (http://technet.microsoft.com/ru-ru/sysinternals/bb897440)

26 NTFS Stream Explorer, a program for working with NTFS streams (http://hex.pp.ua/ntfs-stream-explorer.php)

27 NTFS File Information

(http://forum.farmanager.com/viewtopic.php?t=2050)

28 You can disable the creation of a blocking thread for files in the local editor group policy(gpedit.msc):User Configuration

vatela/Administrative Templates/ Windows components/ Attachment Manager / Deleting information about the zone of origin of attachments.

29 Zone.Identifier stream (http://hex.pp.ua/Zone.Identifier.php)

one or more files" and opening files is blocked. When you open the Properties window in Explorer for a file received from the Internet, the Unblock button appears at the bottom of the General tab and

"Caution: This file came from another computer and may have been blocked to protect your computer" ", button press Unblock removes the Zone.Identifier stream.

Using an Internet browser, download the STREAMS.zip file (you can download any small file, specifying its name in the command below), save it to the root folder of the F: drive, view the contents of the Zone.Identifier stream with the command:

MORE< F:\Streams.zip:Zone.Identifier

Open the Properties window in Explorer (Alt+Enter or the Properties context menu command) for the downloaded file, on the General tab, click the Unblock button, and repeat the previous command in the console.

Create a test file with a command that redirects the text of the text output operator, add an alternative stream, view the result:

ECHO Main text > F:\M.TXT

ECHO Hidden text > F:\M.TXT:Secret.TXT

TYPE F:\M.TXT

MORE< F:\M.TXT:Secret.TXT

An alternative text stream can be loaded into notepad:

NOTEPAD F:\M.TXT:Secret.TXT

Alternative streams can also be created for folders and system files30.

Streams are also used to store extended attributes31.

30 Hidden storage of data in streams of the $Repair file in the system directory $RmMetadata (http://hex.pp.ua/RmMetadata.php)

31 Advanced NTFS attributes and FAT16

(http://hex.pp.ua/extended-attributes.php) 53