SteamPipe: Steam’s new content distribution.

#1
Steampowered

General questions

Q. What’s happening?

We are upgrading TF2, DoD:S, CS:S, and HL2:MP to be delivered through SteamPipe, Steam’s new content distribution. This will change how the game is downloaded, and how it is stored on your computer. The benefits to players are:

Faster and smoother downloads

Faster game boot times and map load times

Easier distribution, installation, and management of mods

Smoother dedicated server distribution and update rollover


Q. Do I need to do anything?

Probably not. A few groups of users might need to take special action:
Authors and users of client-side mods (for example, those that require you to download a .zip and extract it into your installation folder) will want to switch to the new system for distributing and installing these mods.

Users who make custom versions of game files (custom models, custom HUDs, etc) will need to make some simple changes in order to continue using those customizations.

Gameserver operators will need to switch to the new download tool, Steamcmd, and be aware of how the changes to the filesystem affect any customizations they have made. (Please see the separate FAQ for gameserver operators.

Q. What games are being converted?

Here are the games being converted, and their current status:
Day of Defeat: Source Converted.
Half-life 2: Deathmatch Converted.
Counter-Strike: Source Converted.
Team Fortress 2. Scheduled to be converted on Tuesday, April 30.

NOTE: In the remainder of this article, we will use folder names particular to Team Fortress 2 in the examples. The same principles apply to all of the games being converted, so just replace "tf" with "dod", "cstrike", or "hl2mp"

Q. Will I have to download the entire game again?

No. A one-time conversion process will occur, which will take a bit longer than the usual download. You should expect your computer to do a lot of disk work, and you will need enough free hard drive space for about two full copies of the game. (For TF, around 15GB free is recommended.)
With SteamPipe, the shared Source engine files are no longer shared between games. So if you have multiple Source engine games installed, the size on disk will increase by a few GB.

Q. How long does the conversion take?

TF2 can take 20-30 minutes to convert, even on machines with fast hard drives. The other games take closer to 10 minutes or so.

Q. Where did the game move to?

Previously, each user had a folder that contained several game files. For example,

[steam installation directory]/steamapps/[username]/team fortress 2/.

When the game is converted to SteamPipe, the installation directory is shared between users. The directory will probably be something like

[steam installation directory]/steamapps/common/Team Fortress 2/.

Q. What happened to the .GCF files?

SteamPipe does not use .GCF files. Instead, content is packed up into VPK files, which are similar to .zip files. VPK files are used by other Valve games such as DOTA2, CS:GO, and Left4Dead 2. VPK files are also used on the dedicated server.
Customization

Q. Can I still download custom maps and other files from gameservers?

Yes, and we've improved how this works.
Files downloaded from gameservers will go into the tf/download folder. This segregates downloaded files from shipping files and your own customizations.

Q. What will happen to files I already downloaded or customized?

The conversion process can't tell which custom files you downloaded from a server, and which you intentionally customized. To be on the safe side, it will move all custom files into the tf/download folder.
If you have customized a shipping file, however, you will need to move it from the tf/download folder into a subfolder under the tf/custom folder. (This is described in the next question.)

Q. Can I still customize any game file?

Yes. Create a folder such as tf/custom/my_custom_stuff/ and put your custom files there.
For example, you might customize the scoreboard by extracting the resource/ui/scoreboard.res file from the appropriate VPK (using the VPK tool in the bin folder, or a free tool such as GCFScape) as tf/custom/my_custom_stuff/resource/ui/scoreboard.res, and then modifying it. When the game boots, it will see the subdirectory my_custom_stuff and add that folder as a search path. Then your version of resource/ui/scoreboard.res will take priority over the official shipping version of the file.
You can have more than one such subfolder under tf/custom, (for example, you might be working on two different customization projects and don’t want to get the files mixed up), but each folder is another search path added in front of the VPKs will slow down load times a little more. When you’re done tweaking your customization, you can always pack it up into a VPK and delete the subfolder.

Q. Why can't I just edit files directly in the game folder like I did before?

Modifying files directly in game installation folders (tf/resource, tf/materials, etc) won't always work anymore. Here's why.
Previously, when the engine looked for a file, it would first search folders on your hard disk, and then look inside the Steam GCF files. This meant that custom files in the tf folder would take priority over the files in the GCF.
This method had two disadvantages:
It slowed down loading, because the vast majority of files were in the GCFs. Booting the game or loading a level would cause many thousands of operating system calls to attempt to open files, most of which would fail.
Files from many different sources were mixed together. (Files you intentionally customized, files downloaded from gameservers you visited, files from third party mods, and any official shipping games files that were not in the GCF.) Removing a mod was difficult because the files were scattered across several directories. This problem will be exacerbated with SteamPipe, because more game files are shipped loose on disk than was the case when GCF's were used.
The new system searches VPK files first, and then search folders on your hard disk. This reduces load times significantly. But it means that the file in the VPK's we ship will always take priority over loose files in the tf folder.
Users interested in understanding the filesystem search paths in more detail should take a look at tf/gameinfo.txt.

Q. Can custom content be put into VPK's?

Yes! And this is the recommended method.
Installing a mod distributed as a VPK is easy: just copy the VPK file to the tf/custom folder.

Q. How can I create a VPK?

The VPK tool ships with the client, in the bin folder. It's a command line tool, but there is one easy way to use it without any typing. If you drag a folder onto the executable, it will create a VPK with the same name as the folder, with the contents of the folder.

Q. I have a mod distributed as a .zip. The instructions say to unzip it directly into my game folder. Should I?

Probably not. First, check if the mod you are installing is available in VPK format. If so, then download the VPK and put it into your tf/custom folder.
If the mod you want to use is only available in a .zip format, then unzip it into a subdirectory under the game's custom folder. For example, if the .zip contains custom player models that look like (heaven forbid) ponies, and one of the files is materials/models/player/scout/scout_head.vtf, then you might make a dirctory such as tf/custom/i_love_ponies. You should unzip the mod such that the custom scout head texture ends up at tf/custom/i_love_ponies/materials/models/player/scout/scout_head.vtf.
Don't unzip the .zip directly into the custom folder! Make a subdirectory!
Create a subdirectory in the tf/custom folder. For example, tf/custom/lots_of_custom_stuff
Copy the directory structure under tf into the new directory you created. For example, the files that were in the tf/materials/ folder should end up in tf/custom/lots_of_custom_stuff/materials/.

Bugs

Q. Where do I report a bug or give feedback for these changes?

For issues particular to gameservers, please send an email to the HLDS mailing list. (You can subscribe to this mailing list here.)
Otherwise, please communicate your feedback using the github bug database.
Q. What are the known issues?

Known bugs:
The dedicated server might fail to load the Steam libraries the first time you boot it. Booting it a second time will fix the problem.
The SDK and content creation tools have several major problems. Please bear with us, we are going to get these fixed soon. We don't have any intention of abandoning our content creators!

Client beta

Q. How do I get the TF2 client beta?

In the Steam client, right click on the game in the games list, and select “properties.”
Select the “betas” tab.
Select the “steampipe” beta.
Close the properties dialog.
Click “play” to launch the game.
You will be prompted with a dialog prompting you that existing content will be converted to the new format.
The conversion process will take several minutes.
This will move all of your files to the new installation folder (see above) and delete existing .GCF files.
NOTE: In the past we have used an entirely different game for the TF2 beta. (AppID 520.) That beta app is not the same as this beta, and is currently discontinued to avoid confusion.

Q. Can I opt back out of the beta?

Yes, but it is very painful. You will need to re-download the game again, and it will not move your settings files and custom files back to their original location.

Q. Can a beta client connect to a non-beta server?

Yes.
Maps loading faster would be a great thing. Especially in TF2.
 
#9
So basically, they're converting their older games to the system every other developer has been using on Steam for the past couple years.
 
#12
This is actually several years old, they've just been a little slow converting their own software.
I don't believe it is several years old. There are two things at work here:
- Valve converting their games to using the VPK system which has been around since Left 4 Dead (this is the several years old bit).
- Their new download system requires the VPK format, so the old games must be converted to use it.
 
#20
Hey Valve, how about you make it so that I can play a game while it's update is downloading? It's annoying when you go to play a game but you have to wait because there's some huge update that has to download first.
 
#23
GCFs had a good run but if games were actually doubly checking if the file system had the file before loading it from the GCF then they were pointless all along, and the new folder system doesn't seem any more obscure than their old way.

So do newer Source game like Dota 2 and CS:GO already use this?
As has been said, all their games since Left 4 Dead have used it. It was sort of weird, really, they had all these systems set up for sharing content between games and then stopped using it.
 
#24
TF2 boots up instantaneously now, it's kind of amazing.

So I can finally tab out of TF2 while loading a map without the client crashing?
Play in borderless window mode, add -noborder to your launch options and select windowed mode in-game and play at your native resolution. Gives you instantaneous alt-tab goodness with no downside.
 
#28
Hey Valve, how about you make it so that I can play a game while it's update is downloading? It's annoying when you go to play a game but you have to wait because there's some huge update that has to download first.
How could that possibly work for a game like TF2 (or any multiplayer game for that matter). You want them to run servers for different versions of the game?

Play in borderless window mode, add -noborder to your launch options and select windowed mode in-game and play at your native resolution. Gives you instantaneous alt-tab goodness with no downside.
Yea I recommend borderless window too, but some folks can have performance dips with it.
 

Stumpokapow

listen to the mad man
#33
I don't believe it is several years old. There are two things at work here:
- Valve converting their games to using the VPK system which has been around since Left 4 Dead (this is the several years old bit).
- Their new download system requires the VPK format, so the old games must be converted to use it.
July 15, 2011 was the first public release of games on the new content distribution system
 
#34
Hey Valve, how about you make it so that I can play a game while it's update is downloading? It's annoying when you go to play a game but you have to wait because there's some huge update that has to download first.
Not very practical considering how their delta patching works, it's not like PC games of yore where every affected file could be delivered as a whole.
 
#36
How could that possibly work for a game like TF2 (or any multiplayer game for that matter). You want them to run servers for different versions of the game?
There would obviously be exceptions where an update would be mandatory before playing, but I was mainly talking about single player games. When there's an update for a Wii U game, you get the option of downloading the update first, or playing the game with the update downloading in the background.

Not very practical considering how their delta patching works, it's not like PC games of yore where every affected file could be delivered as a whole.
I don't fully understand what you mean, but it's ok. I'll take your word for it.
 
#40
You could upgrade TF2 early about a week ago. It did seem to help boot and map loads.

A word of warning: it breaks your .cfg files and whatnot. You have to reinstall them in the custom folder, in a new folder named whatever and then in another folder like "sounds" or "cfg."


So basically what was once: steamapps/username/Team Fortress 2/tf/cfg/your cfg files.cfg

Is now: steamapps/common/Team Fortress 2/tf/custom/new folder named whatever/cfg/your cfg files.cfg
 
#41
This should also allow us to install those Valve games in our alternate install directories rather than being forced to install TF2 on my SSD when I rarely play it nowadays.
 
#43
You could upgrade TF2 early about a week ago. It did seem to help boot and map loads.

A word of warning: it breaks your .cfg files and whatnot. You have to reinstall them in the custom folder, in a new folder named whatever and then in another folder like "sounds" or "cfg."


So basically what was once: steamapps/username/Team Fortress 2/tf/cfg/your cfg files.cfg

Is now: steamapps/common/Team Fortress 2/tf/custom/new folder named whatever/cfg/your cfg files.cfg
I'm surprised that that Steam Cloud doesn't sync them. I don't think I could ever re-produce my current cfg, which was built up over years of playtime.
 
#45
Wow..... I had probably 16gb (as of yesterday?) before I converted TF2 and now I have 6gb left on my hard drive O_O. At least I'm pretty sure it's TF2 that did that. I don't remember downloading anything big recently and I only have a handful of games installed.


Can we delete these GCF files now? At least the TF2 ones.
 
#47
You could upgrade TF2 early about a week ago. It did seem to help boot and map loads.

A word of warning: it breaks your .cfg files and whatnot. You have to reinstall them in the custom folder, in a new folder named whatever and then in another folder like "sounds" or "cfg."


So basically what was once: steamapps/username/Team Fortress 2/tf/cfg/your cfg files.cfg

Is now: steamapps/common/Team Fortress 2/tf/custom/new folder named whatever/cfg/your cfg files.cfg
thanks