Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide access to the engine for community support #1712

Closed
SamVanheer opened this issue Jun 22, 2016 · 228 comments
Closed

Provide access to the engine for community support #1712

SamVanheer opened this issue Jun 22, 2016 · 228 comments

Comments

@SamVanheer
Copy link

SamVanheer commented Jun 22, 2016

As suggested by @Stenudd, this is a new issue to cover discussion about engine access and upgrades.

Original discussion here: #1685

So far, the following users have shown interest in providing support for engine access:
@tschumann
@johnstt
@Stenudd
@SamVanheer

To summarize: we would like to request access to the engine so we can provide support to the GoldSource community. This would involve both bug fixes and new features to improve functionality.

I can speak from experience here, given that i used to have access when i was on the Sven Co-op team.
Valve requires the engine to be stored in a secure manner, so if access were given it would require a secure repository. They also require the signing of an NDA, meaning no information found in the engine can be disclosed publicly.

Personally i'd like to see the following issues dealt with:

  • Increase engine limits (max entities/models/sounds/events/etc)
  • Bulk data transfer support (similar to Source's network string tables and entity networking, would solve large map cycle issues, late precaching, etc)
  • Elimination of sys_error and host_error causes (e.g. missing files, late precache, host_error triggers a longjmp which can corrupt and leak memory)
  • Fix known issues (e.g. resource lists too large to send all at once cause overflows)
  • Provide features previously limited to Valve games to mods ( e.g. VGUI2, chrome HTML renderer, GameUI)
  • Improved graphical performance (immediate mode support is pretty poor in some driver implementations)
  • New game interfaces to allow the server and client to be notified of events (e.g. map started loading on client and server, changing map, etc)
  • Improved string pool (currently allocates new block of memory for each call to enginefuncs_t::pfnAllocString, can run out easily for larger mods)
  • Unified memory allocator so everything uses the same memory pool (currently malloc/new don't use the engine's heap set by the -heapsize parameter)

There's more, but i can't think of it right now.

If you support this endeavor, and/or want to assist in the event that access is provided, please post your thoughts.
If you have any ideas as to what could be done to improve the engine, please post them as well.

@ghost
Copy link

ghost commented Jun 23, 2016

As the current lead maintainer of Science & Industry, I have a strong interest in this.

  • I have (many times) settled for unsatisfactory solutions to problems because I cannot change the engine or HLDS source code used by my mod

For example, I recently spent a lot of time trying to improve the shotgun in our game, which involved a few performance bottlenecks and engine limits. As far as I can tell, many of these are simple #defines in the engine source code which ought to be reconsidered almost 20 years after Half-Life was developed and released.

In that same thread, I pointed out how the Half-Life engine and HLDS have open-source competition. However, I would prefer the official Half-Life binaries be kept up-to-date and transparent by the people contributing fixes on this Github repo. I cannot in good conscience ask my players to use an 'alternative engine' when the official engine is a closed-source paid product on Steam.

I also tried to learn how sv_clienttrace works as part of some investigation into hit registration and netcode issues in my mod. Unfortunately, I could not find much information in the official channels. Which brings me to my next point..

  • I often look at the Xash3D and reHLDS code to get ideas when debugging these issues, even though I do not use (or trust) those projects.

If the engine and HLDS code were available as open source, I could make better decisions for my mod and contribute improvements to the code powering the official Steam distribution of Half-Life. For example, I investigated and fixed a very strange bug with tripmines (documented here ). My fix applies to the SDK code, but I would be just as willing to share fixes I make to the other components if I had access to them.

There are many more stories like mine, but all the improvements being made are limited to the mods under which they are developed (since this repository is inactive and the Steam distribution of Half-Life is not being regularly updated).

  • I have seen how beneficial a complete open source ecosystem can be with games such as Quakeworld and Quake 3 Arena.

As I explained here, the Quake games are fully available as open source (client, server and engine code). These games are still sold on Steam and elsewhere, but the people purchasing them today are able to benefit from improvements made by open source developers in the many years since their original release. Many successful arena FPS game projects have grown out of the Quake 2/3 source code (OpenArena, ioquake3, Warsow etc) and the most commonly used Quakeworld client/server distributions are all in active development on Github.

I believe that, with only a small investment of management and support from Valve, the people posting on this repo could contribute positively to an open source Half-Life (including access to the engine and HLDS source code).

@tschumann
Copy link

I'm quite interested too. I made and continue to develop Sandpit: http://www.moddb.com/mods/sandpit/
The biggest hassle is the model precache limit of course, but also the inconsistency of it crashing when developer is 1 but silently failing when developer is 0.
The engine's selective ability to load Blue Shift maps is also a bit annoying - it only seems to do it if the game is Blue Shift, otherwise the game freezes.
GameUI being closed source is something I can live with but the inconvenience of not being able to filter what maps show for multiplayer remains - it seems like filters for Half-Life and Opposing Force are hard-coded, and the only alternative is the mp_filter key in liblist.gam which is quite limited. There is also the bug-based alternative of putting single-player maps a .pak file because GameUI doesn't look in .pak files but it's not the best.
Another thing I'd like to do is add support for Dreamcast and PlayStation 2 models, which may be possible but difficult just from client.dll
Also, better community access to the engine would be nice just to see how some things actually work. It's great that with the Linux update for Half-Life a whole lot more of the SDK got documented and released, but as almost none of it is used by vanilla Half-Life it's not entirely clear how to properly use some of the new features. I've been tinkering with IFileSystem009 but some of it doesn't work like you think it should. The ability to better debug some crashes would be nice too - unitialized sizebuf_t doesn't tell me what I'm doing wrong in my mod.

@SamVanheer
Copy link
Author

The engine is full of checks that see if the game/mod dir is a particular one, like czero or cstrike, bshift, etc. That's why it's so inconsistent. The filesystem is actually somewhat buggy. I used it in a separate project to load content and it crashed after removing search paths. I should report that one when i've reproduced it.

@tschumann
Copy link

tschumann commented Jun 25, 2016

Do you know to use AddSearchPath properly? I'm trying to load Opposing Force content and I can't figure it out. I can get it to work but I don't want to change all the models paths to gearbox/models/whatever.mdl when models/whatever.mdl should work.

@SamVanheer
Copy link
Author

SamVanheer commented Jun 25, 2016

AddSearchPath takes 2 parameters: the relative directory and a path ID.
The directory is used to form file paths, while the path ID is used to determine which operations the path is used in.

The path is formed like this:
"<search path/><filename>"

Assuming you're using the file system in a mod, adding the path "gearbox" will work.

The path ID will have to match one used in the engine for it to be used though.

The following path IDs are in use from what i can tell:
ROOT: Half-Life/
BASE: Half-Life/
DEFAULT: a bunch of default paths, not really used directly
PLATFORM: <game dir>/platform
GAME: Game directories, including localization and _addon
GAMECONFIG: Contains game configurations, meaning <mod dir>
GAMEDOWNLOAD: The <mod dir>_downloads directory

It will use the "valve" directory as well to fall back to original content, unless a -basedir command line parameter is specified, in which case the given parameter is used.
The mod dir is determined by using the value for -game. If none is given, it uses -basedir or valve.

If "fallback_dir" is defined in liblist.gam, it also defines GAME_FALLBACK.

If you want to add the Opposing Force directory, you'll have to add its search path. Ideally, adding it to "GAME" should be done since that's where it gets everything else from.

Note that you'll be adding it to the end of the list, so the mod dir and game dir are checked first. Worst case scenario you'll have to wipe all paths and add them again. That's difficult to do given that some are added conditionally based on things like low violence settings and command line parameters.

The engine will search for files by either passing null as the path ID, meaning all paths are searched, or by specifying a specific path ID, in which case only paths with that ID are considered. Whenever you're dealing with content that can be in _downloads, the null path ID should be used, since searching each path ID separately is a major hassle.

Because it searches every path when using null, it will produce duplicate results. That's why the map command and the Options menu models list have duplicates. The same paths are searched multiple times because they're defined in multiple path IDs.

Avoid calling RemoveSearchPath. That's the method that caused crashes for me. I don't know why, but i'm sure that if ran using a debug build or using pdbs the problem should be discovered easily.

Disclaimer: all of this information was determined through reverse engineering. No official documentation exists, so there's no alternative.

@tschumann
Copy link

Hm, I'll try "GAME" - I think I was using "MOD" based on a VDC article...

Also, to add to the list - crashing due to missing .wad files probably shouldn't happen - the engine can certainly handle missing textures; r_shadows could be exposed again (even if only when cheats are enabled) and if you want to print to the console from from some engine callbacks in client.dll then it won't work unless you use GameConsole003.

@SamVanheer
Copy link
Author

I made the same mistake at first. GoldSource and Source use different path IDs for some reason.

I agree with those issues.

@SamVanheer
Copy link
Author

Another reason to provide access: i've built a prototype shader based BSP renderer. It's not complete yet, but it can load and display maps. Because it uses VBOs and shaders, its performance is much better than the engine's immediate mode based renderer.

Repository can be found here: https://github.com/SamVanheer/BSPViewer

If this were integrated into the engine, it could boost graphics performance by a considerable amount. Further improvements in the future could lead to complete support for all rendered objects, including studio models, sprites and particles. This should eliminate any performance issues caused by poor driver support (e.g. AMD drivers).
It shouldn't be too difficult to provide a shader based OpenGL renderer alongside the immediate mode version, though the codebase would be an even bigger mess than it is now.

@SamVanheer
Copy link
Author

I've created a VDC page for GoldSource's IFileSystem interface. You can find it here.

@einfacharthur
Copy link

+1

@Loulimi
Copy link

Loulimi commented Jul 13, 2016

I am strongly in favor of open sourcing Goldsource. The fact this engine powers some of the best games of all time and that modders are still working with it 20 years after its release are proves of the quality of the gaming and development experience with it.
The only reason why I'm reluctant to work more with the Goldsource engine is because it is closed source.

@JoelTroch
Copy link
Contributor

JoelTroch commented Jul 19, 2016

+1 for open-sourcing the engine, a lot of wonderful enhancements, bug-fixes while keeping the legacy of Half-Life is something I wish to see in the future. I agree with the ideas of @Loulimi @tschumann @johnstt and @SamVanheer.

@peon501
Copy link

peon501 commented Jul 22, 2016

+1 I support this idea.

@lcarst
Copy link

lcarst commented Jul 25, 2016

This will be a great step forward for GoldSrc mods if it happens.
It's really about time this happens.

@ssone95
Copy link

ssone95 commented Jul 25, 2016

+1, this idea is great.
It would help us, modders, and people who would like to help to improve the game (since the game is a bit too old now and needs some serious tweaks) and increase its lifespan a bit more :)
👍

@InfcBear
Copy link

+1

2 similar comments
@milutinke
Copy link

+1

@Eraviperoor
Copy link

+1

@dannycolin
Copy link

FYI, you can now thumbs up OP message. That way we aren't annoyed by a ton of notifications for all that +1...

@Skumek
Copy link

Skumek commented Jul 28, 2016

+1, It would help us develope the servers we always wanted to create, and decrease limitations in the engine, fix bugs and performance issues. Even add features such as map instead of radar, HD water and more.

Hope they listen this time, even though in reality i doubt it.

@ghost
Copy link

ghost commented Jul 28, 2016

Maybe if you guys listed some reasons to support this instead of saying +1...

@SamVanheer
Copy link
Author

I could add a new triangle API that actually works in Software and lets you draw sprites with different scales properly. pfnDrawGeneric actually crashes Software mode...

@coreGreenberet
Copy link

+1 as one of the developers of Earth's special forces (esforces.com) I strongly agree that this would help mod developers a lot.

@ghost
Copy link

ghost commented Jul 31, 2016

+1 but does Valve read this repository anymore?

@SamVanheer
Copy link
Author

Something tells me they don't.

@di57inct
Copy link

di57inct commented Aug 1, 2016

@Lamix do u think valve reads anything at all from their clients?

@Stenudd
Copy link

Stenudd commented Aug 1, 2016

@di57inct They have been active in this repository before, no need for a cocky attitude

@ghost
Copy link

ghost commented Aug 1, 2016

yeah, been active in 2013, breaking entire counter-strike.
anyway, nexon has source code and I'm sure they didn't get it for free. It's sad but Valve won't probably give source code of even almost 20 years old engine.

@YaLTeR
Copy link

YaLTeR commented Aug 10, 2016

+1

@a1batross
Copy link

Forget about FreemanValve, they don't care a lot until you don't annoy them(like someone did) or don't trying to sell their code.

That's why ReHLDS and Xash3D still lives.

@SamVanheer
Copy link
Author

It isn't about what the employees want to do, this is a legal issue. The engine is their property, reproducing it is a violation of their trademark. It also contains licensed code and third party dependencies, including Steamworks whose license forbids its inclusion in open source projects.

It would cost them money to open source it, license it or allow you to make your own version, so it won't be allowed to happen. If you're lucky they won't shut you down, but that is a risk that you will choose to make. I won't tell you to do so, and i won't do it myself either.

@ghost
Copy link

ghost commented Feb 23, 2018

Does anyone else find it funny that you guys are discussing all this on Valve's github page ?

If you're really curious, go read some of the law related to computer crimes. Especially in the US, some of it is so broad and vague that just sneezing near a computer can get you jail time. Reimplementing a closed-source commercial product that's still being sold on the world's most popular game marketplace is a bit different, I'd wager..

@dannycolin
Copy link

@johnstt Does anyone else find it funny that you guys are discussing all this on Valve's github page ?

Yep. People still amaze me...

@SamVanheer
Copy link
Author

You don't understand what you're doing. It's still illegal to do this even if you're not using SDK or reverse engineered code.

Plus, you don't know where the GSR code came from. VGUI2 is not a wrapper around VGUI1, it's an entirely different GUI system. It's largely reverse engineered so it's not legal to use.

Xash has questionable status as well, some parts may be derived from leaked code, we don't know that for sure but it seems likely. Valve considers it to be illegal.

If you don't want to do anything illegal then you have to stop working on this now and remove the repositories containing the code.

In any case, you should stop discussing it here. We've established that this will not happen and trying to do it ourselves is illegal, so Valve is well within its rights to lock this discussion. If you continue to discuss it i'll have to lock this issue myself.
I'd rather leave it open in case any meaningful discussion ever takes place however.

@a1batross
Copy link

a1batross commented Feb 23, 2018 via email

@SamVanheer
Copy link
Author

That's still illegal.

@ghost
Copy link

ghost commented Feb 23, 2018

The main thing is that he revealed his motive: he needs 'helpers' because there's an absolute deadline on this wacky half life github project.

If you're convinced this is all legal, go ask Valve's legal department to stamp it.

@jjleebs
Copy link

jjleebs commented Feb 23, 2018

'helpers' because there's an absolute deadline

Unpaid Internship: Patsy

@ghost
Copy link

ghost commented Feb 24, 2018

Why do you not develop Openfl/Lime or C# with Half-Life? Because generator can from source into more end of binary like static and all platforms

OpenFL / Lime has many platforms like Windows, Linux, MacOS, Android, iOS and more more.

Why not we wait longer longer until we have gray hairs. Stop I will kick your butts. If you love Half-Life than you can try to create Haxe OpenFL/Lime or NME etc... It has OpenGL ES3 It works all platforms.

Please wake up VALVE SOFTWARE! Please check good platform generators like Haxe or Mono

Haxe:
OpenFL/Lime or NME generates to all platforms COMPLETED.

Mono:
Bundled and protect executable into native executable
MonoDroid / Xamarin.Android for Android
Xamarin.iOS for iOS Tablets and Phones
Xamarin.Watch for WatchTV
and more ...

Please supervise that!

I argue time because I feel they don't know and force time until very late.

If they won't develop than I will create own or I learn more advanced development.....

@SamVanheer
Copy link
Author

We already know they'll say no, i suggest you act accordingly.

@ghost
Copy link

ghost commented Feb 24, 2018

Let's do a thinking exercise for a moment.

  • Imagine you're Alfred.
  • You have a day job, and work to do at Valve
  • Some person on github asks you to review their entire repo, and it seems to be a legal matter
  • You have no incentive to help them reimplement your game that your company makes money from selling
  • You have a legal team you can forward these things to in less time than it takes to reply on the github thread

What happens next?

@rsKliPPy
Copy link

It also contains licensed code and third party dependencies, including Steamworks whose license forbids its inclusion in open source projects.

I know this doesn't matter for this discussion because there are other legal matters in question, but are you sure that's true? If my understanding of the SDK Access Agreement is right, you are not allowed to distribute the source code, but you are free to use it wherever you want (be it open source software or anything else). Your open-source software doesn't have to have SteamWorks SDK code in the source tree.

@a1batross
Copy link

I suggest some another way of thinking:

  • Imagine some guys periodically asks you about legality, when you intentionally released non-
    encrypted
    DLLs on Windows and Linux builds with full debuginfo.

As I said before: While you doing it just for fun, no one cares a lot, unless you everytime distract some busy people from Valve.

Let's think why GoldSource is great? It's great because it's easy for indie developer to create high-quality game. Why there is no good Source-based indie games? Because it's too complex.

If you want to create a game, you maybe don't need GoldSource at all. You need it's game development flow, where you can create quality game by yourself or really small team. Unkle Mike already tried to make an engine with "Goldsource-like" game development flow, but without GS compability. And he went to situation, where he have a game engine without a game itself. That's why he made partial GS compability. [fat-joke]When some current GoldSourceRebuild-ers just born[/fat-joke]

P. S. While you all here discuss about legality, some people already completely reverse-engineered GoldSource from debuginfo and NetQuake source. I will not post who are these people, because they have GitHub accounts and they will post their project by themselves, when it will be ready enough.

@SamVanheer
Copy link
Author

https://partner.steamgames.com/doc/sdk/uploading/distributing_opensource

It doesn't explicitly cover open source applications that don't have a license, but i would assume that any open source applications cannot contain such code. Maybe i'm wrong about this, but i'd rather be wrong than to get sued.

Making your own copy of GoldSource with Steamworks code in it to authenticate could be used to bypass anti-cheat measures, though i would consider mods accessing Steamworks APIs to be legal since Source mods do just that.

@a1batross
Copy link

a1batross commented Feb 24, 2018

@SamVanheer Warsow is opensource game and it's is intended to have SteamWorks support in Steam release. It's game engine QFusion have some highlights of SteamWorks support(link).

Warsow isn't released on Steam yet, because they don't have developers for Steam release. It's sad.

@rsKliPPy
Copy link

rsKliPPy commented Feb 24, 2018

The Steamworks SDK is incompatible with some open source licenses, which may affect your ability to distribute open source software via Steam.

The key word here is "some". That's because some open-source licenses forbid you from linking to a closed-source object. As far as I know you are allowed to add exceptions to GPL-3.0 that would allow your work and derivatives of your work to link to specific closed-source software. That's what AMXX and SourceMod do. However there are other licenses that would allow you to do it with no additional exceptions, I believe LGPL is one of them.

Btw I'm honestly just interested in what actually is allowed and I'm open to discussion, not trying to be a jerk or correct someone as I myself don't precisely know the entirety of this.

@a1batross
Copy link

  • GPL allows you to link FROM free software TO proprietary, but not otherwise. Like from free & open game to steamworks. Complete solution here is to allow recompile game with or without SteamWorks SDK and allow to use it on Steam release, thus allowing cheats too, lol. :)
  • LGPL itself omits this restriction.
  • AGPL forbids everything than free & open source.
  • BSD-like... it doesn't have such restrictions at all.

@ghost
Copy link

ghost commented Feb 25, 2018

@BlackPhrase THANK YOU FOR SHARING FILES. I miss that - Why Valve Software forces me longer time wow. Thanks I will use with Stream I will replace to Haxe or C# I will tell how do I protect again bad hackers.

@SamVanheer
Copy link
Author

As has been said before you should not discuss this here, this is Valve's issue tracker and what you're doing is not permitted by Valve. We've tried long and hard enough, it isn't going to happen.

I'm closing this issue now, and i hope you will not discuss this any further on any of Valve's issue trackers.

@dannycolin
Copy link

@BlackPhrase congrats...

@ghost
Copy link

ghost commented Feb 26, 2018

@BlackPhrase do me a favour: go up to the top of this issue and read the first few posts.
Do you see the difference?

Thanks for ruining this thread! (although it was already a waste of time to begin with, I suppose)

@a1batross
Copy link

a1batross commented Feb 26, 2018 via email

@BlackPhrase
Copy link

BlackPhrase commented Oct 13, 2018

Alright, so for all this time since February I've been working on reimplementing the GoldSrc engine using the original GPL Id Tech 2 Engine (NetQuake/QuakeWorld/Quake2). It's still rough and ugly as hell, but partially works. The codebase is fully clean and consists only of GPL (or other open GPL-compatible licenses) code. SDK code is also partially reimplemented using the original code (some pieces were manually retyped). Can't say for sure, but the game code will probably also end up being reimplemented (but it either will be ported Quake (QC progs) code or reimplemented HL Alpha level functionality). I will release it on November 8 this year

What will be provided in the initial release:

  • Authentic C code with a lot of wrappers to C++ - one of the goals was to make the codebase look the same as the original GS, by looking at some trace logs it's possible to extrapolate and say that all original Quake code is still in C, all 1998-2002 era code (game interfaces) are also in C and all wrappers are in C++;
  • GameDLL support - support to the point of dlls being playable;
  • ClientDLL support - partial support to be usable;
  • GameUI support - will support original menu code (= HL Alpha) moved to gameui dll;
  • Reimplemented fs module - filesystem code was moved to a filesystem_stdio module (VFileSystem009) and a wrapper for it was added at the engine level;
  • GL Render improvements - QGL integration (gl exports are loaded dynamically, the same as GS does), transparent textures support and such;
  • Reimplemented interface factory code with virtual destructor removed from the IBaseInterface which make modules ABI-compatible between various compilers (but no compatibility with original GS modules);
  • Reimplemented listenserver/dedicated launchers - the code was modified to use IEngineAPI and IDedicatedServerAPI interfaces;
  • vguiwrap - custom intermediate level module to optionally support original vgui1 library (compatibility with original client dlls);
  • Reimplemented pm_shared module - mostly will contain pmove code moved from the engine without any bunnyhop fixes and/or such;
  • User Messages system reimplementation - to add custom game messages;
  • Studio models support;
  • WAD3, SPRITE_VERSION 2, BSP format 30 support - mostly tweaked constants. It probably will temporarily support 29 format again to load the HL Alpha techdemo map later;
  • Partially reimplemented protocol 48 support (excluding resource sending and delta encoding);
  • "SOMETHING ELSE, WILL BE ADDED LATER"

I doubt that I will be working on it after that, so it would be nice if someone could provide some help with stuff, it would be sad if this thing ends up nowhere after all the effort put in it
If the community needs an open-source GoldSrc engine this is pretty much the chance to make it real IMO. As guys at Valve said, they might open-source the engine, but not interested in that, and I'm not interested in making them interested

@SamVanheer
Copy link
Author

SamVanheer commented Oct 13, 2018

Stop posting in this issue, and stop referencing it elsewhere.

@ghost
Copy link

ghost commented Oct 13, 2018

Let's all review some material to avoid getting taken for a ride by this guy!

https://www.merriam-webster.com/dictionary/pied%20piper

https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses

https://github.com/ValveSoftware/halflife/blob/master/LICENSE

https://en.wikipedia.org/wiki/Power_vacuum

http://opentranscripts.org/transcript/fascism-fauxpen-source/

Things to note:

  • Saying "I will release it on November 8 this year" - building hype, or putting the cart before the horse?
  • Code is not already available in a public repo, and yet this project is ostensibly meant for the public
  • Everything is partially implemented and he needs your help (cult leader / con artist behaviour)

This is the type of garbage you encourage by not open-sourcing the engine, and leaving this repo/issue open to rot.

@SamVanheer
Copy link
Author

I can't lock this issue, only people with administrative access to this repository can do that.

@kisak-valve and @Nephyrin if possible please lock this issue.

@ValveSoftware ValveSoftware locked and limited conversation to collaborators Oct 13, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests