Why isn't there a standard way to securely transfer files directly over the Internet?

[gomipile]

At this point, shouldn't there be a way to directly transfer any size of file to a friend over the internet that is well known enough to be a household name? Most of the easy ways require both people to have a free account of some sort, and most of those upload the file to a server so the other can download it. That not only seems inefficient, it also raises privacy concerns. The ones that don't require an account or an upload are old school programs and require a specialized download of some sort, which is a non-starter for my less technical friends.

Back in the day, most of my friends were tech savvy, so it wasn't a problem. But nowadays most of my friends get annoyed if they to go through downloading and setting up a piece of software to accomplish something that'll only be done occasionally.

Also, most of them are not using desktop OSes for whatever requires this. I'm always using a desktop OS and they're usually using a flavor of Android or iOS I'm unfamiliar with. That makes things harder, too.

This seems like something that would have been solved with a household-name solution a long time ago. I would have thought there would be a protocol as well-known as email for one-time point-to-point file transfers that every OS would have its own well-known implementation of?

[gbaji]

At this point, shouldn't there be a way to directly transfer any size of file to a friend over the internet that is well known enough to be a household name? Most of the easy ways require both people to have a free account of some sort, and most of those upload the file to a server so the other can download it. That not only seems inefficient, it also raises privacy concerns. The ones that don't require an account or an upload are old school programs and require a specialized download of some sort, which is a non-starter for my less technical friends.

Unless the folks at both ends of the transfer are installing and using some form of key based encryption file-transfer methodology, those third party file sharing tools are really your best option.


Also, most of them are not using desktop OSes for whatever requires this. I'm always using a desktop OS and they're usually using a flavor of Android or iOS I'm unfamiliar with. That makes things harder, too.

Eh. To be honest though, for 99% or more of stuff you really want to send/recieve between most people, just attaching small files to emails or IMs is plenty sufficient. The more modern model for this is to just have your "stuff" on server clouds anyway, and then just share what you want with others, so there's not really a direct peer to peer transfer going on in the first place (this is certainly how Apple would prefer things). We can debate how "secure" this is, of course.


This seems like something that would have been solved with a household-name solution a long time ago. I would have thought there would be a protocol as well-known as email for one-time point-to-point file transfers that every OS would have its own well-known implementation of?

It's trickier than that though. Direct peer2peer connections are difficult because these days both "sides" are almost always clients of some other hosted connection protocol on which all other communications are piggybacked. Unless both sides have actual servers with real static addresses that are internet routable, you're always going to have an issue asking the question: "is the device requesting/recieving this information actually the device I want to send it to?".

There are some authentication tools (duo, authenticator, etc), that can be used as a challenge/response sort of thing (sometimes, using a physical device for keys). But this still requires some sort of previous communication to establish the initial identity being used. At the end of the day, the first time one device communicates to another and says "I'm device X, and I belong to person Y", you have to trust that it is what it says it is. This is true of more "secure" methods as well. This is one of the reasons why hosted services can often be more secure than p2p methods. If I create an account with a hosting service, and attach a key (or key device) to it for authentication, then it's a good bet that anything I send or recieve was actually sent and recived from/by me. If you do the same, then the host can trust that too. We can then trust that if I grant your account access to something I've uploaded, then you can trust that I sent it, and I can trust that only you can recieve it. Again though, this is all contingent on the assumption that the initial device/key/whatever used to create the account actually belongs to the person you want to share stuff with, and that nothing has changed since then.

And, of course, your precious files are still being stored on someone else's server though. And you have pretty much zero expecation of privacy there. The terms of use for those services usually include some sort of text that says that they can/must share this with law enforcement on request (and often includes other stuff about data mining, trending, "research", etc that can certainly trigger a variety of conspiracy theories if you are so inclined). And in a lot of cases, the hurdle for law enforcement may be "subpeona" and not "warrant" to get at least some of that data (eh, it's an interesting border area where meta-data and actual data intersect). As a general rule, law enforcement can always obtain information about who sent/recieved <something> to/from someone else (the record of the communication, since that's equivalent to phone records, which are owned by the phone company and not you). But they need a warrant to actually read/view what was sent/recieved (which is more akin to a tap on a phone line from a legal point of view).

Honestly? Unless you and your friends set up your own servers (better yet, with your own namespace and not hosted by someone else), with your own keys, and just send stuff back and forth that way, there's not a lot of "security" you can really rely on. I may know a few people who own some class C networks out there, but most don't (they don't sell those to just anyone anymore). So yeah. You can get "close enough" for the most part. Nothing is really 100% secure though. Then there's the "dark web".... duh duh duh!

[NichG]

I mean, putting all the Alice and Bob stuff aside, if you run an ssh server on your machine and give the ip and set up a login and give that information to a friend, they can just use scp (or rsync) to copy a file directly to your machine. If you just need to get a file somewhere and you have a reasonable level of trust with the other person, its a simple solution that's been available for like 30 years...

[gomipile]

I mean, putting all the Alice and Bob stuff aside, if you run an ssh server on your machine and give the ip and set up a login and give that information to a friend, they can just use scp (or rsync) to copy a file directly to your machine. If you just need to get a file somewhere and you have a reasonable level of trust with the other person, its a simple solution that's been available for like 30 years...

Yes, but getting that to happen with a friend who doesn't know what I mean when I ask a question as simple as "What browser are you using?" or "What OS are you on?" would be.... challenging. This is more than 50% of my friends.

[NichG]

Yes, but getting that to happen with a friend who doesn't know what I mean when I ask a question as simple as "What browser are you using?" or "What OS are you on?" would be.... challenging. This is more than 50% of my friends.

Well if you're comfortable with it and they aren't, you could run an ftp server and just give them the URL.

[gbaji]

I mean, putting all the Alice and Bob stuff aside, if you run an ssh server on your machine and give the ip and set up a login and give that information to a friend, they can just use scp (or rsync) to copy a file directly to your machine. If you just need to get a file somewhere and you have a reasonable level of trust with the other person, its a simple solution that's been available for like 30 years...

Well. That's the easy way to do it. But it doesn't match any of the criteria asked for (to be fair, the criteria is inherently inconsistent anyway, but...).

It really only requires one tech savvy person to host this sort of solution. The rest just have to be capable of installing an ssh/scp client, and be able to remember a username/password combo. It does have the benefit from a legal pov, that as a private rather than public service, everything falls into "warrant" territory. But I suppose has the negative that the person hosting is actually legally responsible for all content, so you'd better actually trust your friends that they aren't moving illegal stuff around and using your server in your house to do it.

You know. Just pointing out the one possible pitfall of that. If you have a bunch of yahoos as friends, you might want to think twice about this sort of thing.

[gomipile]

FTP is nice as a techie solution, but I don't see how it helps in this case. I heard that some popular browsers have dropped FTP support. Before, I could have sent someone an FTP link, and they could download a file from an FTP server I set up, sure. But that won't necessarily work anymore.

I'm talking about working with people where asking them to use FTP on the command line is in the non-starter category. Also, how would this work if they're using a smartphone? How would they upload a file to the server?

[NichG]

Well moving down the list, you could just serve the files out over http... As far as upload it'd need at least a little bit of scripting I guess.

But like, this is where we get after discarding the more standard options. The problem isn't the absence of methods, it's the lack of tech savvy. Even if there were a perfect solution, if people never learn to use it, it won't be usable...

So this becomes more like 'why don't average users send each other files securely?' rather than 'why isn't there a way'. Average users will send it in an unencrypted email, and that's that.

[gomipile]

Well moving down the list, you could just serve the files out over http... As far as upload it'd need at least a little bit of scripting I guess.

But like, this is where we get after discarding the more standard options. The problem isn't the absence of methods, it's the lack of tech savvy. Even if there were a perfect solution, if people never learn to use it, it won't be usable...

So this becomes more like 'why don't average users send each other files securely?' rather than 'why isn't there a way'. Average users will send it in an unencrypted email, and that's that.

The file transfer I was dealing with that inspired this thread is too big to receive with Gmail. Probably most other cloud email providers, also.

[TaiLiu]

Your question reminded me of this comic (https://xkcd.com/949/). :smalltongue:

It's a good question. Probably one that might benefit from a historian's response. A possible partial answer is that it's hard to monetarily benefit from something like a P2P file transfer app. Google benefits from us using Gmail and Google Photos cuz they can use our data. Obviously the P2P app would lack that security concern, but the creators wouldn't benefit from us using it for "free."

EDIT: Wait... Are you looking for solutions? Maybe something like file dot love (https://file.love/) would work for your needs? Can't vouch for it or anything.

[halfeye]

I just want to say I never did learn how to use Kermit.

[Telok]

So what's the issue with an encrypted archive file on something like Google drive? Upload, tell them the password, once they have it you can delete it off the service. Sure, technically it's still got a copy out on one of their backup servers. But without a reason for an actual person to look for it, dig it out of backup, and then crack it there's basically no worries. All modern OSs should have access to the basic archiving & encryption you'd need for it. Worse case is you find a free app for them to load up and link them to that when you tell them the password.

[gomipile]

So what's the issue with an encrypted archive file on something like Google drive? Upload, tell them the password, once they have it you can delete it off the service. Sure, technically it's still got a copy out on one of their backup servers. But without a reason for an actual person to look for it, dig it out of backup, and then crack it there's basically no worries. All modern OSs should have access to the basic archiving & encryption you'd need for it. Worse case is you find a free app for them to load up and link them to that when you tell them the password.
It wouldn't have worked in the case that inspired this post. Anything with cloud storage instead of direct connection was vetoed by one of the parties.

The torrent based solutions seem interesting, but apparently can't send to iOS devices sometimes? Also, bittorrent is blocked in some places, so those wouldn't be "this always works" go-to solutions.

[Willie the Duck]

At this point, shouldn't there be a way to directly transfer any size of file to a friend over the internet that is well known enough to be a household name?

I don't have any solutions others haven't mentioned, but wanted to comment on this. Printers have been in common public use a lot longer than ubiquitous internet (and thus need to transfer private data over it), and yet getting them all to conform to universal drivers and protocols has been a never-ending fight (with there being X standard options, and whenever someone tries to create a 'universal standard,' there now becomes X+1 standard options). Some times there just isn't a standard choice for a thing and every time someone tries to make a new standard, it just becomes another of the many options, no one of which comes to the fore.

[Murk]

I don't have any solutions others haven't mentioned, but wanted to comment on this. Printers have been in common public use a lot longer than ubiquitous internet (and thus need to transfer private data over it), and yet getting them all to conform to universal drivers and protocols has been a never-ending fight

Yeah, but if I understand gomipile correctly: "printing" is a standard, household concept.
If the situation is "I have a file on my pc that I want to have on paper instead", then printing it is the first thing that comes to mind.
Sure, every printer works differently and there are different brands, but everyone knows printers. All offices have a printer, and lots of households.

Similarly, if the situation is "I want to send someone a message from my pc", then email used to be the immediate common solution. Nowadays chat applications might have replaced email (depending on your culture/environment), but for a long time it was email. Maybe different email services and brands, but all companies had email addresses and most individuals did, too.

But if the situation is "I have a file on my pc that I want to send to someone else", then there isn't one common solution that immediately comes to mind.

[Batcathat]

But if the situation is "I have a file on my pc that I want to send to someone else", then there isn't one common solution that immediately comes to mind.

Maybe it depends on who you ask, but I feel like "attach it to an email if it's small, use a service like Google Drive or Dropbox if it's big" is very much the go-to solution. Obviously there are downsides to it as mentioned in the OP, but I don't think that many people care about them in most cases.

[Murk]

Yes, but those are three very different solutions! One is email, one is through an application that you install linking pcs, one is by uploading it and then having the other person download it.
And XKCD's solution of just chucking it on a usb stick is entirely different again.

Compared to that - all email is basically the same, and all printing, and all chat functions (once internet took over from things like sms).
So I share gomipile's curiosity why none of the file transfer solutions ever got the upper hand over the others.

[Chronos]

Your goals are fundamentally in conflict. You want very strong security, and you want it to be usable by non-tech-savvy people. By far the weakest point in security is people doing stupid things, because they're not tech savvy. You can try to get a third party to do as much of the security part as possible, to decrease the opportunity for non-tech-savvy people to do stupid things, but that will always depend on trusting the third party.

[HeraldOfExius]

Your goals are fundamentally in conflict. You want very strong security, and you want it to be usable by non-tech-savvy people. By far the weakest point in security is people doing stupid things, because they're not tech savvy. You can try to get a third party to do as much of the security part as possible, to decrease the opportunity for non-tech-savvy people to do stupid things, but that will always depend on trusting the third party.

I think this is going to be the main reason there isn't something that meets gomipile's criteria. Your average non-tech-savvy user isn't going to be concerned about things like the fact that a cloud storage service puts their files somewhere else on someone else's machine(s). Hence why everyone uses things like email, Google Drive, or Dropbox to share files over the internet. And given that these already fill the "transfer files" requirements for 99% of common users, there isn't really an incentive for anybody to create a new protocol and go through the effort of getting it integrated into every major OS (the other large reason).

Take FTP, for example. It (or one of its more secure offshoots) would meet the requirements, but only if it were available and active by default in every OS. Based on a bit of googling, Windows has FTP client/server capabilities built in, but requires a dive into your settings to enable them, while Android and iOS do not have FTP built in, but it's implemented in a lot of third party file system apps. Neither of those is particularly non-tech-savvy-friendly, but it is an implemented standard. But, personally, I'm glad from a security perspective that not every device out there is a complete FTP client/server package, because that many possible P2P connections between file systems in the hands of ordinary users is not something I ever want to have to deal with, going back into Chronos' point about security and non-tech-savvy users.

[Storm_Of_Snow]

Not over the internet, but if the recipients are that tech non-savvy, how about a USB stick and the postal system?

There's actually some uses (admittedly massive data sets like Hubble images) where it's quicker to do that than try and pull it electronically.

[gbaji]

It wouldn't have worked in the case that inspired this post. Anything with cloud storage instead of direct connection was vetoed by one of the parties.

And that's the problem though. If cloud storage solutions are off the table, then you have to have some sort of p2p solution, and that will always require some level of technological sophistication on the part of those using them. If that's off the table then...


The torrent based solutions seem interesting, but apparently can't send to iOS devices sometimes? Also, bittorrent is blocked in some places, so those wouldn't be "this always works" go-to solutions.

What's funny is that the problem is usually not the actual transfer of the files, but getting them from/to the device that is formatted to use it. I can trivially move hundreds of GB of data in a wide variety of forms around to various sites around the world whenever I want. But heaven help me if I need to share a picture I took on my phone in a specific remote meeting tool on my laptop. The higher "level" a tool operates on, the more difficult it actually is to move things in/out of it.


Not over the internet, but if the recipients are that tech non-savvy, how about a USB stick and the postal system?

There's actually some uses (admittedly massive data sets like Hubble images) where it's quicker to do that than try and pull it electronically.

Here's the thing though. Usually if you have an actual need to move large files around, you probably have tools for doing just that. Email works for anything 10MB or smaller, which covers pretty much any standard images, screenshots, and most file attachments. The only stuff I usually see that is too large to handle this way are software installation/data files. And those usually start out hosted on a server configured for download (usually some ftp/http thing).

I guess the other question one might ask is "what data are you copying around, and why are you doing it"? A lot of the time, folks are trying to do things that they maybe shouldn't be doing in the first place (copying software from person to person instead of downloading it from the actual source, copying music/videos in the same way, etc). So yeah. You kinda can't be surprised that the folks who make money on you going through them to get this sort of stuff aren't going to build easy to use methods to circumvent them. Go learn how to do this if this is what you want to do. There are absolutely ways to do things like this if you don't want to have to work with/through some one else's service. But all of them require a certain amount of know-how.

[OracleofWuffing]

I feel like I'm inviting a joke (https://xkcd.com/1782/) at the expense of my age here, but does DCC over IRC count?

I get it's not the correct solution, as you'll probably still need to have people download an IRC client of some kind and set it up, and whether or not setting up IRC is user-friendly is kind of subjective in today's day and age. (I mean technically you could always roll your own IRC client to make it exactly as user-friendly as you want, what could possibly go wrong?) It's still a type of a standard that is... Apparently? Still in use today. Somehow.

[Rockphed]

I feel like I'm inviting a joke (https://xkcd.com/1782/) at the expense of my age here, but does DCC over IRC count?

I get it's not the correct solution, as you'll probably still need to have people download an IRC client of some kind and set it up, and whether or not setting up IRC is user-friendly is kind of subjective in today's day and age. (I mean technically you could always roll your own IRC client to make it exactly as user-friendly as you want, what could possibly go wrong?) It's still a type of a standard that is... Apparently? Still in use today. Somehow.

Do people even still use slack? I admit that my employer uses "microsoft teams" and accomplishes most of the same functions that slack did at a previous job, and I haven't used slack except at the one job (partially because I am a luddite), so I am a poor sampler of what gets used.

[Manga Shoggoth]

Do people even still use slack? I admit that my employer uses "microsoft teams" and accomplishes most of the same functions that slack did at a previous job, and I haven't used slack except at the one job (partially because I am a luddite), so I am a poor sampler of what gets used.

Amazingly, yes. I'm currently working with a number of projects that use Teams AND Slack - My company and the main clients use Teams, and the rest of the subcontractors involved use Slack. (I'm not sure why, but I suspect its because it's easier to set up slack accounts than it is to deal with the security implications of letting non-vetted contractors on the internal networks and the effort involved in maintaining said accounts.)

This leaves me in the slightly ridiculous situation where I have three Teams accounts and two slack accounts (and that's before we factor in the two additional equivalents that my employer used to use internally before going to Teams).

[Smoutwortel]

Because the current runners of the internet are large cooperations who want everything to go through their servers, so they can spy on it and large files make that hobby expensive.

[kyoryu]

Because it's too hard of a problem to solve with the general level of tech knowledge the typical user has, with too little upside.

"I want to securely transfer files" is, for the most part, reasonably achieved by sending an encrypted zip file/etc. via email, or storage provider. The additional value of "this encrypted file only exists on the source and destination machines" is, for the typical user, very low.

From there, though, to get to machine-to-machine transfer without any middleman requires a bunch of stuff - especially around networking, where NAT is standard. Just getting your IP address is tricky for most people, and then working around a scenario where one or both sides is firewalled or NATed gets into really tricky spaces for the non-techy.

[NichG]

Because it's too hard of a problem to solve with the general level of tech knowledge the typical user has, with too little upside.

"I want to securely transfer files" is, for the most part, reasonably achieved by sending an encrypted zip file/etc. via email, or storage provider. The additional value of "this encrypted file only exists on the source and destination machines" is, for the typical user, very low.

From there, though, to get to machine-to-machine transfer without any middleman requires a bunch of stuff - especially around networking, where NAT is standard. Just getting your IP address is tricky for most people, and then working around a scenario where one or both sides is firewalled or NATed gets into really tricky spaces for the non-techy.

Worth pointing out that that is a totally artificial and constructed difficulty though, a consequence of the industry decision to make NAT standard.

[Chronos]

If nothing else, to do a direct peer-to-peer transfer without any third party involvement, you need to have both parties online at the same time. There are ways you could do this without both people having to be online at once (one of them setting up some sort of server that's always running, and then leaving their computer and internet access on), but then you're back to someone doing something technical.

And, of course, if you don't trust a third-party hosting site to keep your data private, do you trust a third party to create whatever other software tools you're using for the direct transfer? It wouldn't be hard for a software provider to include a backdoor in something like that to let them access your data.

[Bohandas]

I've gotta agree that it seems weird that there isn't an existing system for this given that it could theoretically be done by combining two technology from way back in the 1990's. Back in the 1990's there were answering machines, which could pick up and record an incoming phone call, and there were also fax machines, which could transmit an image over the phone lines, so theoretically you could send an image file by recording the fax beeps instead of stupidly routing them directly to a printer. A similar effect could be achieved for other file types by replacing the fax machine with a dial-up modem)

There seems to have been a general proliferation in useless middlemen since then however, as exemplified by the fact that answering machines have all been replaced by voicemail services. There should be an app that does the job of an answering machine - picking up the call and recording a message locally - but there's not, I've checked multiple times

[gbaji]

I've gotta agree that it seems weird that there isn't an existing system for this given that it could theoretically be done by combining two technology from way back in the 1990's. Back in the 1990's there were answering machines, which could pick up and record an incoming phone call, and there were also fax machines, which could transmit an image over the phone lines, so theoretically you could send an image file by recording the fax beeps instead of stupidly routing them directly to a printer. A similar effect could be achieved for other file types by replacing the fax machine with a dial-up modem)

This is exactly what does exist right now. There's nothing preventing you from hooking up computers at two different points, attaching both to an ISP, and then communicating directly between them. But yeah, instead of connecting a modem or fax at each end, you just install some software that allows for secure login and file transfer directly from one to the other. This pretty much requires, however, that at least one person install a linux OS on a computer and use it as a server (you can actually do this with windows, but yuck!). It's not that difficult to do. Finding your IP address isn't that difficult either, but unless you pay for some sort of colocation service, or enterprise level ISP service, your address will change periodically (and may still change even with an "enterprise" service, depending on the specifics). Some of this depends on how your ISP is set up though.

Assuming you do obtain a routable IP address, the rest is just installing the OS, and setting up ssh/scp/sftp services on it (which is often as easy as "turning it on"). Then setting up accounts and creating passwords and then handing them to your friends. The problems with this is that if you want this to be relatively unhackable (I said "relatively"), you really want some sort of firm firewall rules to block addresses other than your friends from accessing your system (no system is fully defended if it's sitting out there "bare naked" on the internet for any length of time). Which of course is a problem since their addresses will change frequently as well. But this is no different than your "hook up my fax/modem to the phone line" solution. Anyone who has your phone number can connect, right? Same deal. Basic process hasn't actually changed much at all.

And yeah, the other problem is that if you want to actually have any other computers at your home, attached to the same ISP connection, you must use a switch/router, which puts all the devices behind a private network, which isn't routable anymore (including the server you just set up). Which now gets into more complicated work of setting the server up "in front" of your switch/router, and using it as a gateway system. Which, yeah, gets into some more complex networking skills that 99.9% of folks out there just aren't going to grok at all.

Or you can just use a third party service. Or stick to using email attachments, and accept a 10MB filesize limitation. But yeah, if you want to do this, and you really really don't want to risk some third party seeing your precious data (you can always encrypt the files btw), then you need to crack open some books and start learning. There's a ton of sites on the interwebs that can tell you how to set this sort of stuff up, but some of the advice is terrible, and some is just poorly written, and all of it requires quite a bit of learning curve.

[Telok]

Wait a sec. Just use email anyways. Encrypted archive, there's super simple (ok there was, I haven't needed to do this in 20 years) software to break into 8mb chunks & reassemble the archive at the other end. Takes a bunch of emails but if you're really worried don't send them in sequential order and include a dummy package or twenty. Instructions on order of assembly are passed by phone call.

Should be able to be handled by any decent archive software (probably built in still) and the rigamarole makes people feel they're doing special fancy secret message passing.

[Bohandas]

This is exactly what does exist right now. There's nothing preventing you from hooking up computers at two different points, attaching both to an ISP, and then communicating directly between them. But yeah, instead of connecting a modem or fax at each end, you just install some software that allows for secure login and file transfer directly from one to the other. This pretty much requires, however, that at least one person install a linux OS on a computer and use it as a server (you can actually do this with windows, but yuck!). It's not that difficult to do. Finding your IP address isn't that difficult either, but unless you pay for some sort of colocation service, or enterprise level ISP service, your address will change periodically (and may still change even with an "enterprise" service, depending on the specifics). Some of this depends on how your ISP is set up though.

Assuming you do obtain a routable IP address, the rest is just installing the OS, and setting up ssh/scp/sftp services on it (which is often as easy as "turning it on"). Then setting up accounts and creating passwords and then handing them to your friends. The problems with this is that if you want this to be relatively unhackable (I said "relatively"), you really want some sort of firm firewall rules to block addresses other than your friends from accessing your system (no system is fully defended if it's sitting out there "bare naked" on the internet for any length of time). Which of course is a problem since their addresses will change frequently as well. But this is no different than your "hook up my fax/modem to the phone line" solution. Anyone who has your phone number can connect, right? Same deal. Basic process hasn't actually changed much at all.

And yeah, the other problem is that if you want to actually have any other computers at your home, attached to the same ISP connection, you must use a switch/router, which puts all the devices behind a private network, which isn't routable anymore (including the server you just set up). Which now gets into more complicated work of setting the server up "in front" of your switch/router, and using it as a gateway system. Which, yeah, gets into some more complex networking skills that 99.9% of folks out there just aren't going to grok at all.

Or you can just use a third party service. Or stick to using email attachments, and accept a 10MB filesize limitation. But yeah, if you want to do this, and you really really don't want to risk some third party seeing your precious data (you can always encrypt the files btw), then you need to crack open some books and start learning. There's a ton of sites on the interwebs that can tell you how to set this sort of stuff up, but some of the advice is terrible, and some is just poorly written, and all of it requires quite a bit of learning curve.

My point is that there doesn't seem to be an app for it, either on the Google Play Store or on F-Droid, and the way you deacribe it it sounds like it still requires an extra device other than the sender, the receiver, and the ISP.

What I'm imagining at the moment - and there's certainly better ways to do this, but just for the sake of example - is that this would replace your phone app - you wouldn't be able to answer the phone anymore because it would interfere - and the way it would work is that when you wanted to send a file it would convert it to modem noise and dial the person who you wanted to receive it. Then the copy of the app on their end would answer the phone automatically, convert the modem noise back into a file, and automatically save it to their downloads folder, no servers or password cloak-and-dagger required

EDIT:
Similarly, I would replace voicemail with a program that automatically picks up after 3 or 4 rings, plays a locally saved prompt, and then saves the message to your phone as an mp3. As stated above though, there doesn't appear to be any app that can do that.

[DavidSh]

I really don't like the idea that anybody with a smart phone connection could put arbitrary files onto your phone. There's enough of that sort of thing already, but this is just another security hole. I would insist on authorizing any incoming transfer before it starts.

[Bohandas]

Hmmm... I guess maybe it would have to be two apps, and the reciever app would be blocked from making outgoing calls

[gbaji]

My point is that there doesn't seem to be an app for it, either on the Google Play Store or on F-Droid, and the way you deacribe it it sounds like it still requires an extra device other than the sender, the receiver, and the ISP.

This took literally 5 seconds on google to find:
https://play.google.com/store/apps/details?id=com.estmob.android.sendanywhere&hl=en_US&gl=US&pli=1

Or is that not what you are looking for (no clue. Haven't used this app, don't know how it works). I was talking about how to transfer files from one device to another.

There are dozens of similar tools you can use. But yeah, you are trusting that the app creator, and or transfer methodology isn't doing something along the way. That's not something you can eliminate though. Not unless you more or less "roll your own".


Similarly, I would replace voicemail with a program that automatically picks up after 3 or 4 rings, plays a locally saved prompt, and then saves the message to your phone as an mp3. As stated above though, there doesn't appear to be any app that can do that.

Why do you want to do this (and btw, this is not about file transfers, but I get it)? The only difference is where the recording is stored. The advantage of voicemail is that it's stored on a server, so you can access it from any device, and it doesn't go "poof" if you drop your phone in the toilet or something. Also, it works even if your phone is off, ran out of power, is out of service area, etc. It's kinda the whole point of doing it that way.

Literally the only reason to even care about this is if you are so concerned that someone might say something on your voicemail that will get you into some sort of legal trouble or something. But again, if we're just talking about having some means to communicate without being listened in to, there's a very simple method: Use a code. Send text messages, use a code. Done. Ask your local drug dealer how they do this. It's not that hard (I'm by no means advocating for illegal activity here btw).

I have issues with cloud based storage for a lot of things and a lot of reasons. Voicemaili is somewhere very near the absolute bottom of that list though.

Oh. BTW. This is *exactly* how email works too. When you configure your email client, you point it at an smtp server, right? That server actually stores your incoming email for you (you know, just like a voicemail box). When you connect with your client, you download a copy of that email to your local device to read it. It's still on the server. You can use commands to clear email from the server if you want, but the point is that it's always first sitting on that server. Back in the day, we called this a "mail spool" (it actually sits in a directory called "spool", and why linux systems have a /var/spool/mail directory, among others). Those mail files used to be just plain text files. Anyone who had root on the actual mail server could read them. It was not uncommon for those files to become corrupted, and the filter tools would not work, and users could not recieve mail. A common fix was for some admin (like me) to log into the mail server, find the text file containing the email, and edit it (with like vi). Usually there was some set of garbage characters, or a malformed message at the top of the file (you got pretty good at seeing the pattern of characters that indicated start and stop points of each individual message within the text file).

My point is that pretty much every communication you do over any sort of network results in various copies of what you are communicating sitting around at various points along the way. Some of these represent actual security risks. Most are just trivial and unimportant (or you just accept it as a cost of doing what you are doing).

Worrying about voicemail? Why? There are vastly bigger fish to fry here.

[HeraldOfExius]

I really don't like the idea that anybody with a smart phone connection could put arbitrary files onto your phone. There's enough of that sort of thing already, but this is just another security hole. I would insist on authorizing any incoming transfer before it starts.

Preferably with a spam filter so that when your contact information invariably gets onto a mailing list you don't have to manually refuse hundreds of obviously malicious files.

[Bohandas]

Oh. BTW. This is *exactly* how email works too. When you configure your email client, you point it at an smtp server, right? That server actually stores your incoming email for you (you know, just like a voicemail box). When you connect with your client, you download a copy of that email to your local device to read it. It's still on the server. You can use commands to clear email from the server if you want, but the point is that it's always first sitting on that server. Back in the day, we called this a "mail spool" (it actually sits in a directory called "spool", and why linux systems have a /var/spool/mail directory, among others). Those mail files used to be just plain text files. Anyone who had root on the actual mail server could read them. It was not uncommon for those files to become corrupted, and the filter tools would not work, and users could not recieve mail. A common fix was for some admin (like me) to log into the mail server, find the text file containing the email, and edit it (with like vi). Usually there was some set of garbage characters, or a malformed message at the top of the file (you got pretty good at seeing the pattern of characters that indicated start and stop points of each individual message within the text file).

My point is that pretty much every communication you do over any sort of network results in various copies of what you are communicating sitting around at various points along the way.

That's the whole point of this discussion. All of these systems are needlessly rube-goldbergian


Preferably with a spam filter so that when your contact information invariably gets onto a mailing list you don't have to manually refuse hundreds of obviously malicious files.

Perfect. That would entirely eliminate of my original proposal's two flaws. (And I'm sure the plaw of it taking up the actual phone could be eliminated by routing it through some other system or, worst case scenario, by simply adding a second line


The advantage of voicemail is that it's stored on a server, so you can access it from any device

You mean that hackers, jealous significant others, meddling patents, your service provider, the NSA, and the KGB (or whatever they're called now) can access it from any device

[Vahnavoi]

The standard way is a hardline between two terminals and nothing else.

If by internet, you mean "over the worldwide web", rather than just TCP/IP protocol , you are doing some heavy abuse on the word "directly".

Beyond this, the reason for lack of a standard way is because "secure" is often at odds with "usable". The natural trend is towards multiple different standards depending on how much crap end users are willing to tolerate and which kind of security risks they actually face.

[gomipile]

The standard way is a hardline between two terminals and nothing else.

If by internet, you mean "over the worldwide web", rather than just TCP/IP protocol , you are doing some heavy abuse on the word "directly".

Beyond this, the reason for lack of a standard way is because "secure" is often at odds with "usable". The natural trend is towards multiple different standards depending on how much crap end users are willing to tolerate and which kind of security risks they actually face.

Why would I mean "over the worldwide web" when I said "Internet?" By "over the Internet" I mean between two computing devices which are connected to the Internet, each of which can be anywhere in the world an unfiltered Internet connection is possible.

By "secure" I meant "using strong end-to-end encryption."

[Rockphed]

Why would I mean "over the worldwide web" when I said "Internet?" By "over the Internet" I mean between two computing devices which are connected to the Internet, each of which can be anywhere in the world an unfiltered Internet connection is possible.

By "secure" I meant "using strong end-to-end encryption."

The strongest end-to-end encryption is a pile of random bits the same length as your file to securely transfer. The sender does some reversible math (probably an xor, but I am sure there are other options) between the pile of bits and the file to be sent, the receiver does the inverse. For slightly less strong encryption, use fewer truly random bits. It doesn't have to be random bits, but it does need to be secret between the two of you. If you decide to do something bizarre like combining the data with the text of an 1812 version of the bible in Latvian (complete with a weird forward about how the monks in Finland will rue the day), it is unlikely that anyone will be able to parse out what either your secret key or your data is, but if it gets out then everyone will know. The random bits just need to be truly random, not just pseudorandom like most of the "random" numbers computers use.

[gbaji]

That's the whole point of this discussion. All of these systems are needlessly rube-goldbergian

Explain "needlessly". How exactly would you allow someone to send email to someone else (or anything similar)? I mean, sure, your "solution" of having some sort of hard connection via phone lines or something, and using modems can work. But it's not terribly portable or usable. And is just as susceptible to interception as modern email systems (moreso really). Your own lack of understanding of how telecommunications systems actually work doesn't make "wishfull thinking" a valid alternative here.

And the whole "stuff gets copied" is how electronic files work. Period. It's not like a physical letter that is carried from point to point. You are transmitting data across a wire. There's nothing "physical" involved here. It's the same for anything that is "transmitted". Think how radio works. The DJ at the station puts a record on the player. It's hooked up to a transmitter. He's not transmitting the physical vinyl record, right? He's transmitting a "copy" of it (in this case patterns of RF, that are picked up by antennas and converted back into sounds). In that case, anyone with an antenna tuned to the correct frequency/amplitude will pick it up and hear a "copy" of the music being played by the DJ.

Computer networks transmit across wires instead of through the air, but it's the same concept. You create patterns of signal, and send it down some sort of wire. It must be routed, which requires specific protocols for that, and it must be picked up at each point, copied, and then sent on to the next (eh, I'm super oversimplifying packet handing methodologies here). You also must have some way to determine where the endpoints actually are, and how to get the signal to go between them. And deal with error correction (what happens when the signal gets garbled, which happens all the time).

The point is that unless your "network" more or less consists of a single wire, strung directly from your house to your friends house, with nothing in between, there's no way to avoid this. And absolutely none of it is "unnecessary".



You mean that hackers, jealous significant others, meddling patents, your service provider, the NSA, and the KGB (or whatever they're called now) can access it from any device

And what exactly is your alternative proposal? Again, once you get more complex than "two tin cans with a string", you're going to have these issues.

You don't honestly think that even old school phone lines were any different, do you? You do understand that before they had computer managed switching systems, they had literal people running switchboards, right? You'd tell them which number you wanted to connect with, and they'd connect physical wires do to this. And they had earpieces and were listening in, pretty much the whole time. It was trivial to record people's conversations back then, if someone nefarious wanted to.

There really is no "good old days" when it comes to telecommunications technology. There are methods to actually do things today that are much more secure than ever before, but it requires that folks at both ends of the conversation have some savvy (and honestly, just some easy to use encryption tools).


Why would I mean "over the worldwide web" when I said "Internet?" By "over the Internet" I mean between two computing devices which are connected to the Internet, each of which can be anywhere in the world an unfiltered Internet connection is possible.

Those are often used as synonymous terms though. Well. Technically "web" often refers to http based content, but at the end of the day, that's just one type of "thing" that could be at the other end of a IP based connection.

And btw. There is no such thing as an "unfiltered internet connection". Does not exist. Again, unless you are just directly connecting wires between your own computers, you are *always* going through someone else's network to get from point A to point B. I mean, you could be talking about higher level software usability tools (like web browsers), but even using other tools, you're still traversing the exact same connections, using the same routing protocols, and the same naming/addressing protocols to get there.

How do you think "the internet" actually works? Even "the dark web (duh duh duh!)" uses the same protocols. The only difference is that they use unadvertised name servers and routing hosts to bounce packets around. They still go through the same switches, and the same backbone conduits, and use the same packet methods (sorta) as everything else. And are exactly as susceptible to being picked up along the way if someone is really trying to do so. It's honestly not terribly hard to noodle out when specific message forms are being sent across a network to a non-adverstised system, what that system is doing and acting as. Anyone with a computer terminal on a network segment can simply "listen in" to all traffic on the segment. Takes no special ability or access to do this (well, except having something plugged in right there).

Unless I'm really not understanding what you mean by "unfiltered". Maybe clarify that? If you mean "not going through someone else's system and being processed" then you're in for a rude awakening.


By "secure" I meant "using strong end-to-end encryption."

Which is perfectly valid. Again though, the actual transmission methodology is the same one being used for everything else. Encrypting the contents of a file is a very good idea. But it does not do anything about all the other layers involved in a communication "across the internet" that absolutely allows the entire thing you just sent to be copied by any number of people. You're just relying on them not wanting to spend the time/effort/cost to decrypt your message out of all the other messages out there.

If you really don't want someone to hear what you are saying, then don't transmit it in any way at all. Period. Anything transmitted is capable of being copied. Always. Unless you physically own and control/secure every foot of wire between the endpoints of the communcation, this is always the case.

[Bohandas]

And the whole "stuff gets copied" is how electronic files work.

There's a difference between something being immediately retransmitted by a router or a repeater and it being needlessly saved in some secondary location


And what exactly is your alternative proposal? Again, once you get more complex than "two tin cans with a string", you're going to have these issues.

Save the pbone message directky to the phone instead of storing it on a voicemail server. We've been through this.


EDIT:
Anoher thing I've noticed that seems needlessly rube-goldbergian, why do so many screen casting devices need access to your local wi-fi? Your phone has a transmitter and a receiver and so does the casting device, and they're in the same room of the same building, so why the heck can't they communicate directly? (edit: I've seen several that claim to be able to do this, but so far none of the ones I've bought have worked; all of them asked to connect to my local wi-fi and wouldn;t function without doing so)

[tyckspoon]

That's the whole point of this discussion. All of these systems are needlessly rube-goldbergian


A huge portion of the technologies and basic conveniences that make up modern life are based on somebody's hacked together poorly optimized proof-of-concept demo, which they assumed that once they got serious money in it and could get somebody interested in developing it they'd remake it to be better or the initial approach would only be used until somebody thought of a better way to do it.

.. and then the proof of concept becomes the standard, warts and all, and everybody adopts it, and by the time it is widely known that it's kind of a terrible horrible no good way to do the thing -everybody is doing it- and changing off of it would require coordinating millions of people to shift to something new and cost billions of dollars to implement.

[gomipile]

And btw. There is no such thing as an "unfiltered internet connection". Does not exist. Again, unless you are just directly connecting wires between your own computers, you are *always* going through someone else's network to get from point A to point B. I mean, you could be talking about higher level software usability tools (like web browsers), but even using other tools, you're still traversing the exact same connections, using the same routing protocols, and the same naming/addressing protocols to get there.



Sorry. "unfiltered internet connection" was shorthand meant to avoid details that would probably be against the forum rules.

Suffice to say, I meant an internet connection that allows most normal traffic. One that isn't subject to [REDACTED POLITICAL REASONS] filtering of the sort that most of us here probably don't have to live with.

I didn't mean the sorts of filtering that most, say, Usonians and Canadians experience daily with their Internet connections.

If you have a better shorthand for that then please, suggest it and let us use it in this thread henceforth.

[halfeye]

There's a difference between something being immediately retransmitted by a router or a repeater and it being needlessly saved in some secondary location

There is no difference that the sender or receiver can detect, the router saving itself a copy would be completely invisible at both ends. I don't know how many routers the typical internet connection goes through, could be 5, could be 50, probably isn't 5 million, any one of those could in theory make a copy, and none of the rest would know.

[gomipile]

There is no difference that the sender or receiver can detect, the router saving itself a copy would be completely invisible at both ends. I don't know how many routers the typical internet connection goes through, could be 5, could be 50, probably isn't 5 million, any one of those could in theory make a copy, and none of the rest would know.

There is a philosophical difference between it maybe but probably not being saved by a middleman, who might not even get to see every packet of the overall transfer, and it definitely being saved by some cloud provider.

Also, an actual factual practical concern is that the possible file size WILL be limited by the amount of storage space allowed by a cloud storage provider at whatever subscription tier is used. However, the practical limits on direct transfer are time, transfer speed, and patience. Even if you're only "getting close" to using up the storage limit of a cloud provider, that adds the extra step of having to manage the cloud storage carefully so as not to overload it.

Sure, this is just pushing tradeoffs around, but there are lots of technologies where different approaches with different tradeoffs are available for various applications and preferences.

[Bohandas]

There is no difference that the sender or receiver can detect, the router saving itself a copy would be completely invisible at both ends. I don't know how many routers the typical internet connection goes through, could be 5, could be 50, probably isn't 5 million, any one of those could in theory make a copy, and none of the rest would know.
I'm not talking about routers making a copy, I'm talking about the needless extra step of it being held in cloud storage instead of being sent directly* to the recipient's phone.

*directly as in without stopping[i] at any intermediate point, Going through however many routers and however many repeaters but without ever stopping to be deliberately, and as part of the protocol, held on some intermediate drive that the recipient must actively contact.

[i]At the absolute most it only need be held only long enough for a connection to the recipient to be established, but even this is not strictly becessary if you're willing to re-send the file


Also, an actual factual practical concern is that the possible file size WILL be limited by the amount of storage space allowed by a cloud storage provider at whatever subscription tier is used.

Not to mention the fact that you generalky have to pay for their stupid middleman in the first place in order to make any non-trivial use of it. That in itself is a dealbreaker.

Additionally, things stored on a cloud server may have to abide by terms of service. That's a dealbreaker as well. And some providers snoop through people's private files to make sure they're abiding by the TOS (and/or as part of some bogus pecksniffian crusade against one or more of the Four Horsemen of the Infocalypse to make themselves look good), and that's definitely a dealbreaker

[Vahnavoi]

Why would I mean "over the worldwide web" when I said "Internet?" By "over the Internet" I mean between two computing devices which are connected to the Internet, each of which can be anywhere in the world an unfiltered Internet connection is possible.

Because when you are talking about global connected network, you are talking of the world-wide web. That's not direct. Not even close. Anything that's not two terminals connected by a line involves going through routers and quite likely your message being transformed from hardline to wireless signal several times over, and each such point is one where the signal can be intercepted.


By "secure" I meant "using strong end-to-end encryption."

I was not contesting your use of the word "secure". I was contesting your use of "direct". The entire point of strong encryption is that you can pass your message through an indirect channel like the internet and be reasonably sure the contents remain safe even if your signal gets intercepted.

The actual answer to your question remains the same.

[DavidSh]

Because when you are talking about global connected network, you are talking of the world-wide web.

I guess the original definition of the world-wide web as the globally distributed hypertext document, using HTTP, and including many implicit or procedurally generated pages, is dead. This originated some years after the Internet was first created, as the large network using the Internet Protocol (IP). Just like "literally" as anything other than a vague intensifier is dead.

[Bohandas]

Just like "literally" as anything other than a vague intensifier is dead.

And also "very" and "really" which also mean "literally"

[gomipile]

Because when you are talking about global connected network, you are talking of the world-wide web. That's not direct. Not even close. Anything that's not two terminals connected by a line involves going through routers and quite likely your message being transformed from hardline to wireless signal several times over, and each such point is one where the signal can be intercepted.

I use the internet in ways that don't involve a browser, even a hidden one where the browser engine is used as a back-end way of retrieving and possibly displaying content. From your earlier post:



If by internet, you mean "over the worldwide web", rather than just TCP/IP protocol , you are doing some heavy abuse on the word "directly".

It seemed like you were using a more or less normal definition of the World Wide Web. I guess that may not be the case?

Colloquially, most techies I talk to associate the World Wide Web with the worldwide accessible collection of documents that can be accessed with HTTP in a browser. Technically, I know that current technical usage includes many document types and means of access, though AFAIK, always involves documents accesses with something like a browser.

A direct transfer of a file that doesn't make that file available as part of the World Wide Web wouldn't be "over" the World Wide Web, I would think.

[Vahnavoi]

The real point is that it makes a difference whether you're talking of a specific messaging protocol versus the infrastructure.

In a technical sense, if you're talking about just a protocol, you can send a file without using the world-wide web. You'll still be sending that file over the physical world-wide web.

[gomipile]

You'll still be sending that file over the physical world-wide web.

If it's sent in a way that it's never addressable except locally inside the filesystems at each end, then it's never part of the World Wide Web.

Say I'm sending it, then if I transfer it in a way that gives it a public address the recipient can use to access and then download it, then there might be an argument that it is part of the World Wide Web at that time. But, if I use an application that negotiates the transfer in some other way, say by the receiver agreeing to accept a connection and allocate the required storage, and then the application pushes a bunch of packets to the receiver without the file ever being addressable to pull from any filesystem, then the file is never "on" the World Wide Web by any useful definition I've seen.

[Vahnavoi]

You are adamant to ignore the actual point in favor of pedantry, aren't you? The physical world-wide web is not the messaging protocol, it's the infrastructure that relays your message. Because this process is not direct, there are multiple points and multiple ways to intercept your message. That's what it physically means to transfer anything over the internet. Whether your message gets put on the internet in the sense of being accessible via a
browser (etc.) is immaterial.

Encryption as a form of security exist because the way of sending the message piggybacks on other people's machinery and there's no easy way to tell who might be eavesdropping or how many copies of your message are left in the system as it passes. This thing (f.ex.) Bohandas wishes for, a way for a signal to pass through the entire system without ever stopping, ever being held on an intermediate drive, so on and so forth? There are technological and infrastructural reasons why this doesn't, and likely can't, happen over the physical world-wide web. You want that, you have to build your own physical network, either via hardlines or some kind of directed energy impulse. Which messaging protocol your network uses, at that point, is a curiosity.

[halfeye]

You are adamant to ignore the actual point in favor of pedantry, aren't you? The physical world-wide web is not the messaging protocol, it's the infrastructure that relays your message. Because this process is not direct, there are multiple points and multiple ways to intercept your message. That's what it physically means to transfer anything over the internet. Whether your message gets put on the internet in the sense of being accessible via a
browser (etc.) is immaterial.

What you described above is the internet:

https://en.wikipedia.org/wiki/Internet


This article is about the worldwide computer network. For the global system of pages accessed via URLs, see World Wide Web.

...

The Internet carries a vast range of information resources and services, such as the interlinked hypertext documents and applications of the World Wide Web (WWW), electronic mail, telephony, and file sharing.

...

The origins of the Internet date back to the development of packet switching and research commissioned by the United States Department of Defense in the late 1960s to enable time-sharing of computers.[2] The primary precursor network, the ARPANET, initially served as a backbone for the interconnection of regional academic and military networks in the 1970s to enable resource sharing.

https://en.wikipedia.org/wiki/World_Wide_Web


The Web was invented by English computer scientist Tim Berners-Lee while working at CERN. He conceived it as an information management system using several concepts and technologies, the most fundamental of which was the connections that existed between information.[8][9][10] The first proposal was written in 1989,

This may be ancient history that you don't want to know about, but some of us lived it, and it's not something we're going to forget just because you don't care.

[HeraldOfExius]

Additionally, things stored on a cloud server may have to abide by terms of service. That's a dealbreaker as well. And some providers snoop through people's private files to make sure they're abiding by the TOS (and/or as part of some bogus pecksniffian crusade against one or more of the Four Horsemen of the Infocalypse to make themselves look good), and that's definitely a dealbreaker

While I generally agree with wanting such services to not look through their clients' files, the unfortunate truth is that they may be liable for the content of said files depending on current local laws (which can be different for the client's, service provider's, and server location's various "locals") and whether or not the judge deciding the case when something happens is feeling agreeable. Unless we get a definitive, global law, I don't think this issue will go away. It probably won't go away even if we could get such a law, but at least for now I can see why some service providers would want that in their TOS. We just have to not use them.

[Vahnavoi]

You are simply restating information in a way that does not move the discussion forward in any manner. I already explained what the point was multiple times. Let's do it once more for clarity: if you want to talk about the specific messaging protocols used, then "the internet" is machines using TCP/IP and WWW is HTTP (etc.). But if you want to instead talk about the physical system of interconnected machines, then "global" is "world-wide" and "network" is"web".

Notice that I'm using "physical" instead of "literal" because you pedants had to pre-emptively ruin that for me. If you'd not, the entire point could be collapsed to "in addition to the technical definition of world-wide web, there is a literal world-wide web of machines to run it".

And I maintain that the answer to the title question is based on physical realities of that world-wide web.

[Chronos]

If I'm talking about pets, I could use the word "cat" to mean "furry four-footed animal", so long as I'm careful to define my terms. But even if I'm careful to define my terms, using the word "cat" to mean "furry four-footed animal" will only cause confusion, because that's not what everyone else uses the word to mean. And so I don't use that word that way.

Similarly, you can patiently explain that you're using the word "World Wide Web" to mean "the Internet", but even if you explain that, it's still needlessly confusing, because that's not what everyone else uses the word to mean. And there's already a perfectly good word for "the Internet", so there's not even any benefit to using the word "World Wide Web" instead.

[gomipile]

You are adamant to ignore the actual point in favor of pedantry, aren't you?

It's not just pedantry. Whether a method is on the WWW or not is a useful distinction, and your wrong definition makes that distinction impossible to discuss with you until you check yourself and correct yourself.

If a document/file does not have a URL, it isn't on, part of, or anything about the World Wide Web. Files, documents, other data, traffic, connections, etc. on our over the Internet that don't have URLs are not on the Web.

[Bohandas]

Similarly, you can patiently explain that you're using the word "World Wide Web" to mean "the Internet", but even if you explain that, it's still needlessly confusing, because that's not what everyone else uses the word to mean.

Actually in common speech they are interchangable, even if in technical parlance they are apparently not


While I generally agree with wanting such services to not look through their clients' files, the unfortunate truth is that they may be liable for the content of said files depending on current local laws (which can be different for the client's, service provider's, and server location's various "locals") and whether or not the judge deciding the case when something happens is feeling agreeable. Unless we get a definitive, global law, I don't think this issue will go away. It probably won't go away even if we could get such a law, but at least for now I can see why some service providers would want that in their TOS. We just have to not use them.

Which further just drives my point home more that you shouldn't trust cloud servers. Even if the company wants to be trustworthy and decent there may be a law that says they can't be, and that they have to snoop like a bunch of creeps.

[kyoryu]

Worth pointing out that that is a totally artificial and constructed difficulty though, a consequence of the industry decision to make NAT standard.

I mean, sorta? Otherwise you'd have to give out a bunch of addresses to people, which would really suck with IPv4 space.

Plus, using NAT routers acts as a pretty good default firewall for lots of bad stuff pinging you.

NAT is a pretty good solution for the majority of problems that the majority of people face. It does create some problems in opening direct peer-to-peer connections without a middleman as a broker, though.



Which further just drives my point home more that you shouldn't trust cloud servers. Even if the company wants to be trustworthy and decent there may be a law that says they can't be, and that they have to snoop like a bunch of creeps.

Which is why you should do your own encryption of any data you want to be secure. That's not actually that difficult, for any level of security the average person is likely to need.

[NichG]

I mean, sorta? Otherwise you'd have to give out a bunch of addresses to people, which would really suck with IPv4 space.

Plus, using NAT routers acts as a pretty good default firewall for lots of bad stuff pinging you.

NAT is a pretty good solution for the majority of problems that the majority of people face. It does create some problems in opening direct peer-to-peer connections without a middleman as a broker, though.


It's sort of an implicit assertion about 'how the internet should be used' - customers are there to receive information and services, not to provide or produce it. When that assertion becomes a default that other people are deciding systematically across an entire industry, well, it creates problems like what the OP is experiencing. There are things you should be able to do which have been made artificially hard because people assume 'you won't want to use it for that'.

[gbaji]

I'm not talking about routers making a copy, I'm talking about the needless extra step of it being held in cloud storage instead of being sent directly* to the recipient's phone.

It's not "needless" though. Doubly so if we're talking about cell phones. What if the phone on the other end is off, or busy, or disconnected from the network for some reason? What do you do with the message? Cell phones (and other devices like tablets) move around. They connect to different wi-fi and cell connections all the time. Knowing how to "send it directly" is not always possible at any given point in time. So yeah, the most convenient way to do this is to have each device act like a client that connects to a server, which holds the data/messages/whatever for delivery.

I've already stated, a couple times now, how you can get around this if you really want to. Have a dedicated network connection with a static routable IP Address on each "side". Then you can communicate directly. The moment you are plugging multiple devices into a private (or public) network, which in turn has its own route to a service of some kind (cell or ISP), which in turn then connects to other name service/rounting services to know "where I am", you are kinda stuck being a "client" of that service and having to work through the limitations of that method.

Yes. You sacrifice your own ability to privately send message back and forth. But the other 99.999999% of the time, this is more than made up for by the convenience of being able to walk around anywhere on the planet and send/recieve messages from/to anyone you want.



It's not just pedantry. Whether a method is on the WWW or not is a useful distinction, and your wrong definition makes that distinction impossible to discuss with you until you check yourself and correct yourself.

If a document/file does not have a URL, it isn't on, part of, or anything about the World Wide Web. Files, documents, other data, traffic, connections, etc. on our over the Internet that don't have URLs are not on the Web.

You are "technically" correct. But for the conversation we are having (how to send data files securely from point A to point B), it's a differnence without an actual "useful distinction".

Go look at the url in your browers window. Heck. Right above us here. See that line: https://forums.giantitp.com/newreply.php?p=25730753&noquote=1

Break it down. The forums.giantitp.com is an "internet address" (technically a fullly qualified domain name, or FQDN for short). It consists of a hostname "forums", and a domain "giantitp.com". That address tells any packet where to go. You will find the same format for email (hostname.my.domain.name.<com/edu/mil/org/etc>. These are name to routable address references. A name server will find the actual address for these FQDNs. In this case it's:

Non-authoritative answer:
Name: forums.giantitp.com
Address: 172.67.193.125
Name: forums.giantitp.com
Address: 104.21.57.233
Name: forums.giantitp.com
Address: 2606:4700:3037::ac43:c17d
Name: forums.giantitp.com
Address: 2606:4700:3036::6815:39e9


Which is cool, cause we got a mix of ipv4 and ipv6 addresses in there.

The "https://" is a header used by the program you are using on your computer (presumably the url bar in a web browser), which indicates that it should send its packets to a port on the computer on the other side based on that info (which for http will typically be 80 or 443, but other ports may also be used for other bits as well). The point is that what you are making a huge detail about is literally just a text format used in the client application.

The stuff following the FQDN is what is processed by the receiving computer. It could be a path to data, or some other string that is interpreted by whatever program is listening to the specified port on the specifiied computer at the other end.

Another common syntax used for IP connections is FQDN:<port num>. Kinda depends on the user level tool being used. The point is that from a packet level communication pov, all you are really doing is telling the system that "I'm sending data to <hostname.domain> on <port number>, with <set of other options>, and <heres my data>. On the remote system, there is a program that listens in to that port on that system (they aren't physical ports btw, just programmatic things), and the data is sent to that program for processing. That program knows what to do with that data. The point is that the concept of "web traffic" versus "all other traffic" is absolutely irrelevant to every step between point A and point B. The computer at point A knows that what kind of data it's sending and formats it that way. The recieving computer at point B is told what kind of data it's receiving, and processes it. No one in between cares.

I can literally set up a completely different program on port 443 on a system, and process "web traffic" using that program instead. I can totally mess with people and assign ports to different listening programs as I wish.

If I want to send stuff via say scp, I might use this syntax:

scp /path/to/local/data <username>@<hostname.domainname>:/path/to/remote/data

Notice that since I'm using the "scp" program, it knows what port to send the data to (port 22), just as your web browsers knows which port to send its data to). I'm indentifying the exact same "hostname.domainname" syntax to actually tell the computer where to send the data. And in this case, I'm sending additional information about what user to log in as on the remote system, and what files are copying from my computer, and where I'm putting them on the remote computer.

The point is that from "the internet" point of view, both communications, and connections are functionally the same. And yes, you may wish to make a distinction between runnning some process via activation on a web page, versus connecting directly, but again, that negates what the OP seemed to be asking for "an easy way to do this for non-techincal folks". Web formats were created to simplify things, so that you didn't have to know much about what you are actually doing. You just load up a web page, and the computer on the other end processees things based on what buttons you click there. You can actually put a web front end on just about anything you want (and trust me, I've seen just about anything used this way).

So yeah. From a "how do I send files securely and directly" point of view? The distinction is somewhat irrelevant. Saying "I don't want to send it over the world wide web", could mean "I don't want to send this from a web client", um... but that's really failing to grasp the underlying commmunication protocol. Lots of "web based transfers" actually use ftp/sftp/etc. And guess what? the web part is just a convenient front end for ease of use. You can literally also just use the ftp program to do this directly (though you sometimes may need to hunt for the actual location of the files you are looking for).

What many people call the "world wide web" is really just a pretty graphical cover on top of "the internet". Not a whole lot of difference really. Except that while everything with a pretty web front end is "on the internet", not everything on the internet has a "pretty web front end". The actual rules and methods for getting packets of data from one point to another? No difference at all. You're doing the equivalent of complaining that you can't drive your car to your friends house without having to deal with traffic signals along the way, but instead of recognizing that this is a feature of "driving on roads shared with other traffic", you're insisting on blaming this on the naviation system in your car. As if, there's some magical usability layer whereby removing the "ease of use" of a turn by turn naviation system (equivalent to an "easy to use web front end") will allow you to bypass the actual intersections, stop signs, traffic signals, and other "rules of the road" along the way.

That's just not the way it works. And no, nothing is "needless". Every single thing about IP based communications is actually very well thought out and has a very specific reason for being there. If you think it's needless, it's because you haven't actually thought about the need.

[gbaji]

It's sort of an implicit assertion about 'how the internet should be used' - customers are there to receive information and services, not to provide or produce it. When that assertion becomes a default that other people are deciding systematically across an entire industry, well, it creates problems like what the OP is experiencing. There are things you should be able to do which have been made artificially hard because people assume 'you won't want to use it for that'.

Service providers use NAT because there aren't enough addresses to go around. It's a techincal solution.

You are using just one of many things that NAT does. It's not like NAT was created by folks who wanted to control other people's data. It's a method of redirecting network traffic within a private net, so that you can pass data back and forth without someone "outside" your own set of rules knowing what's going on.

NAT is literally what allows every single private user to connect 8 or 10 different devices onto a single wi-fi router and have that all "work". It's not some evil nefarious scheme. Yes. As a consequence of that convenience, it does mean that you can't use those client systems as "servers" on the internet. But that's, again, nothing nefarious. It's a limitation of the routable addresses available for use, relative to the massively higher number of actual physical devices connected to the internet.

And yes. It does lend itself to a "model" in which cloud services come into use. I could build my own home network server, then connect all my devices (let's say the controllers for my AC mini-split wall units). I could then build a gateway server and set up a routable address for it, then write a front end "home control panel", to control these devices, using the private net to send/recieve information. Or... I can just use the vendor provided cloud solution, connect the little devices to my home network wi-fi, have them route as normal private net clients out to the service (and configure them to my account), and then configure an app on my phone to also connect to the same account on the same service.

I don't need to direct connect to my home network. I just change a setting on the cloud app, and that gets sent to the cloud server, and the controllers in my house, which are also connected to that server, see the change and send the corresponding commands to the AC unit(s), and I arrive home to find the AC is on, and set to the temperature I set, and I'm nice and comfy. Ease of use. That's it. Could I do this "the hard way". Sure. Is it worth my time? Not even remotely. Pick your battles, and this isn't one worth fighting.

EDIT: Oh. And the "cloud method" is actually, ironically maybe, more secure. NAT means that the clients are making esentialy "per connection" rules. They only "exist" network wise in the context of that one connection being made, and being initiated from the client inside a private network (ie: my house). To enable direct connection, I would have to actually open up those devices to *everything* on the internet, so that I can connect to my home network devices remotely. That's incredibly dangerous. Unless you really really really know how to secure a network, this is probably a "really bad idea"(tm).

[NichG]

Service providers use NAT because there aren't enough addresses to go around. It's a techincal solution.

You are using just one of many things that NAT does. It's not like NAT was created by folks who wanted to control other people's data. It's a method of redirecting network traffic within a private net, so that you can pass data back and forth without someone "outside" your own set of rules knowing what's going on.

NAT is literally what allows every single private user to connect 8 or 10 different devices onto a single wi-fi router and have that all "work". It's not some evil nefarious scheme. Yes. As a consequence of that convenience, it does mean that you can't use those client systems as "servers" on the internet. But that's, again, nothing nefarious. It's a limitation of the routable addresses available for use, relative to the massively higher number of actual physical devices connected to the internet.


I remember when NATs started to become common, the main arguments were security and preventing threat surfaces for users' own good/preventing botnets, etc. Dynamic IPs solve the same technical problem as NATs do from the provider side, and from the customer side if you want to connect 8 devices you can get a router that you control, rather than renting a router/modem from the provider with locked settings. For a period of around 15 years it was standard and easy for me to run an SSH server on my home computer, have it update the dynamic IP with a domain service (and later, just push the IP to a place I could look it up using a cron job that ran once a day), and basically run experiments/do work/access files/etc while travelling around with only a laptop for a thin client. Running my own Minecraft server or things like that was likewise pretty easy.

I can still do that, but the number of hoops to jump through has increased steadily over time.

So when the OP is noting 'hey its hard to do this now, why is that?', the fact that NATs became standard practice and basically unavoidable unless you pay a premium is a valid answer to that. You can argue that NATs are worth it on the balance, but you shouldn't downplay the downsides of that choice or pretend like it was always this way and its strictly necessary for this to be how internet access is organized.

[gbaji]

I remember when NATs started to become common, the main arguments were security and preventing threat surfaces for users' own good/preventing botnets, etc. Dynamic IPs solve the same technical problem as NATs do from the provider side, and from the customer side if you want to connect 8 devices you can get a router that you control, rather than renting a router/modem from the provider with locked settings.

Yes. But the "server" is still using NAT, even if you are not aware of it, when using dhcp granted private net addresses. This happens every time you put a wi-fi router into your home network. You maybe get a set of addresses running from 192.168.0.1 through 192.168.0.254, right? But "upstream", there's an ISP owned server that hands out an address via DHCP to your modem first. That's one address that it can communicate with. That wi-fi router takes that address from the ISP. It then hands out an arbitrary number of private net addresses to everything connected to it. Every single time one of those devices initiates a connection "out" of the network (crosses the ISPs network), that router uses NAT to assign an new "port@host" number to that connection. It then sends it out on its own "port@host" value as the "S" (source), to the "D" (destination). When a reply packet comes back, it's sent to the port/host on the router, which then looks up the NAT table and finds the actual port/host to send it to on the private network.

This is how you can just plug in a dozen devices onto a wi-fi router you purchased at Walmart for $60 and it "just works". But the consequence of this is that you can't initiate a connection to those devices from outside that network. They don't have findable routable addresees on the internet. The only outward facing address is the one that your ISP handed via dhcp to your modem/router in the first place, and that incomming connection has no way to actually tell that first "hop" in your network which of the hidden devices on your private network to connect to.

So, by necessity, anything using NAT is more or less stuck using "outbound only" connections. Now, if you are clever, you can write your own NAT rules, and literally tell your "first hop" that connections on "port X" go to "port Y, on Host Z". That's how you actually set up this sort of thing. And yeah, if you actually only have one ssh capable host on your network, you can just tell the NAT to send all packets on port 22 on the gateway host to port22 on the ssh server you have set up.


For a period of around 15 years it was standard and easy for me to run an SSH server on my home computer, have it update the dynamic IP with a domain service (and later, just push the IP to a place I could look it up using a cron job that ran once a day), and basically run experiments/do work/access files/etc while travelling around with only a laptop for a thin client. Running my own Minecraft server or things like that was likewise pretty easy.

Yup. But for the same reason (limited number of routable addresses and an ever increasing number of stuff wanting them), a lot of ISPs actually use a layer of NAT inside their own networks as well. So you don't even get a routable internet address anymore. You are basically already sitting inside a "private network" inside the ISPs network, and you get an address from that non-routable range, and once again, only outgoing connections can work.

You can get routable addresses sometimes (depends on the ISP, often also depends on how much you pay). But it's by no means a guarantee. And this isn't some kind of evil nefarious thing. It's literally a case of "we only have X number of routable addresses, but we have X*500 customers".


So when the OP is noting 'hey its hard to do this now, why is that?', the fact that NATs became standard practice and basically unavoidable unless you pay a premium is a valid answer to that. You can argue that NATs are worth it on the balance, but you shouldn't downplay the downsides of that choice or pretend like it was always this way and its strictly necessary for this to be how internet access is organized.

It's not really a matter of "worth it on balance". It's "only a small percentage of people would ever be able to connect to the internet" if we didn't use it. And, as with any case of scarcity, who got access would be based on who could pay for it.

"Back in the day", there were lots of addresses available. I know people who owned (may still own) their own class C networks back then. That's just not the case nowadays. In theory IPv6 should/could fix a lot of these issues, but despite being an accepted protocol for a very long time, a lot of the internet just doesn't use it. Or if it does, most end users aren't aware of it, so it doesn't help them much anyway. Do this test. Log into your wi-fi router, and see what address it got from the server. Odds are, it's an IPv4 packet (in standard aaa.bbb.ccc.xxx format).

And at the end of the day, it's irrelevant anyway. The OP wanted something "easy to use". Obtaining and then using your own routable address to communicate between end points is no more easy or hard (tech wise) today as it was "back in the day". If the OP isn't willing to set up their own server to allow for this communication, and then teach the others how to connect to it, and use it, then the availability of an address to "direct connect" is kinda moot, right? Even if we allow for or enable some sort of "global routable addressing methodology", if the user isn't willing to use it, then it isn't useful to them. I mean, technically, that's kinda what we have with cell phones right now anyway. They identify themselves in a unique way to the cell system, which in turn transfers the connection onward within the net (btw, using a system very similar to NAT). It's how we make two-way phone conversations in the first place.

It's also how IM clients on phones work. Again though, there is always a server somewhere processing the communication, but that's because IMs are considered "non-interactive" communication (like email). The other side doesn't have to be immediately available for the message to be sent. It's why you use IMs instead of trying to make a direct call. It's also why you should always text people when they are in emergency zone instead of tying up lines trying to call them. SMS protocol actually fits bits of the messages in between the (much larger) bandwidth segments used by direct two-way phone conversations. So if there's an earthquake or something, and you can't get through on the phone, just send a text. It'll get through. Eventually. Pro-tip, I guess.

Could we set up something for direct connection for data instead of just voice between cell phones and other similar devices? Sure. But you run into the same issue of network efficiency.

[kyoryu]

All of what gbaji said.

I'll also point out that commercial internet is sold on a different set of assumptions than residential.

Specifically, residential internet assumes that downloads are either going to be moderate-usage steady (streaming) or super-bursty. IOW, when you buy 500mbps or 1gbps of bandwidth or whatever, they do not expect you to be using it at that level all of the time - and if you look at commercial internet (where they do assume that), you can see what the prices would be if they did.

And since handing out multiple IPs is really only necessary for actually setting up servers, and servers tend to suck up all the data given teh chance, setting up servers is generally against policy as well. And since those are the primary issues with NAT routers, it's not a problem they really want to fix.

That's also part of the reason why most internet speeds are asymmetrical - very few consumer use cases require more than 50mbps up.

[crayzz]

Which further just drives my point home more that you shouldn't trust cloud servers. Even if the company wants to be trustworthy and decent there may be a law that says they can't be, and that they have to snoop like a bunch of creeps.

As has been mentioned, a "direct connection — no intermediate storage" solution is kind of a nightmare for a variety of reasons since phones and personal computers just do not have reliable uptime the way dedicated servers do, just as a result of how they're commonly used.

That said, the way to handle this would be a mixture of end-to-end encryption and adopting the whatsapp/signal model where the file is removed from the server once delivery is confirmed. You can do your own encryption on top of that if you're worried about the data being forcibly accessed via warrant and in case the devs built a back door into their encryption, but unless the warrant is timed perfectly, its likely the data the warrant is looking for isn't even available anymore.

Signal, for what its worth, allows files up to 100MB, which is larger than most emails allow (unless they're cheating via a secondary cloud storage service) AND the files don't stick around on their servers.

[halfeye]

I halfway mentioned Kermit, I used it once (or twice?) with someone telling me "put this here, and that there", it looks as if it ought to be the system to use if it's still going and is capable of sending big enough files.

https://en.wikipedia.org/wiki/Kermit_%28protocol%29

On the other hand I did once wire up a connection between a NewBrain and an ST over RS232, to save my basic programs (probably no more than three?), that can also work, if you can write to and read characters from a port, but it's not for big files (USB would be the modern equivalent, but I don't know whether that works).

[kyoryu]

As has been mentioned, a "direct connection — no intermediate storage" solution is kind of a nightmare for a variety of reasons since phones and personal computers just do not have reliable uptime the way dedicated servers do, just as a result of how they're commonly used.

That said, the way to handle this would be a mixture of end-to-end encryption and adopting the whatsapp/signal model where the file is removed from the server once delivery is confirmed. You can do your own encryption on top of that if you're worried about the data being forcibly accessed via warrant and in case the devs built a back door into their encryption, but unless the warrant is timed perfectly, its likely the data the warrant is looking for isn't even available anymore.

Signal, for what its worth, allows files up to 100MB, which is larger than most emails allow (unless they're cheating via a secondary cloud storage service) AND the files don't stick around on their servers.

Yeah, that's a good pattern.

End-to-end encryption and manual deletion (encrypted file on gdrive) is more than good enough for most people. End-to-end plus automatic deletion is more than good enough for the vast, vast, vast majority of cases.

Eventually you get to the point of just shrinking the window to access already-encrypted data, which starts to require really, really dedicated attacks. And if you're going over the internet, your data will always be on someone else's hardware for some small period of time.

If you really, really, really want secure transfer? Encrypt it and stick it on a USB/SD card.

[NichG]

I wonder that there aren't (or maybe there are) internet services where you can basically create a virtual network that gives you static IPs within the sphere of that virtual network, that function in all ways like a static IP would but only within that 'neighborhood', and where the user-end software would treat it like IP tunneling locally - so you could run your SSH server and, for anyone logged into the same shared neighborhood it'd be completely transparent...

Even if you have a NAT, as long as you maintain a constant open connection to the neighborhood server you could receive notifications of incoming connection attempts and negotiate those attempts via that one intermediary. I wonder how inefficient it would end up being.

And as far as costs go, call me unsympathetic but I've had the experience of having my internet bill go down by 25%, my speed go up by 1000%, and my downtime reduce by 5x by moving from an urban area in which Comcast has an infrastructure monopoly to a rural area with a different local provider. So while I buy that users trying to get 100% of their promised bandwidth breaks the pricing models, I think there's also a lot of give for people to be able to run their own servers without bringing the whole system down.

[kyoryu]

Sounds basically like a VPN.

[NichG]

Sounds basically like a VPN.

Well a VPN that allows spontaneous incoming connections from users of the VPN though.

[kyoryu]

Well a VPN that allows spontaneous incoming connections from users of the VPN though.

That would be most "actual" VPNs. Most of the ones that non-corporate folks use are just ways to make it seem like you're in a different place.

[Whoracle]

That IS a VPN. Pure and simple. But you still need a server for that, and trust a service if you don't want to setup such a thing. Early examples of such a thing include Hamachi, which we used for LAN-Only games over the internet. Hamachi is now an adware-infested hellhole, but the concept remains.

Problem is that with the VPN you'd still need name servers and the corresponding server software on the end devices so ppl can exchange the files.

And if you're capable of running a VPN server/service for friends and family, along with the neccessary support (hint: not that hard, but still not exactly easy), you're more than capable of running a Linux Box at a hoster of your choice, with encrypted hard drives and an instance of nextcloud (even easier).

Seriously, the smallest Hetzner Box you can get in the US is € 5.18 / mo, which nets you 40GB of storage, so ~25GB after OS and stuff. SSL Certs are free with Let's Encrypt. And with Docker the setup is even easier, since you don't even need to configure a webserver. And the thing has native apps on all major platforms, both mobile and stationary.

[NichG]

€5.18/month is a lot for something you could do by logging into your router once though...

[gbaji]

€5.18/month is a lot for something you could do by logging into your router once though...

Depends on your ISP. If the ISP hands out a routable address, and you know how to build/secure a gateway box, including your own custom NAT and IPForwarding rules (or are ok with only having one computer on your home network, which I suspect most are not), then yeah. You can do this for free. A lot of ISPs these days are going to require that you pay for some form of business account in order to get an actual routable address, and that'll cost more money (but then you "own" it, so that's good).

The negative with the hosted server addresses mentioned above is that, once again, you don't actually own the servers. You are basically renting space that you slice up into a VM/container/docker/whatever. All the data is still technically on disk owned by someone else. So... read the terms of service carefully.

There are certainly some methods to use VPN to do this. Again though, you are relying on someone else's service to route your connections, but in this case you *can* actually have what will at least appear to be a "routable address" for other folks to use, which can be directed to your own computer. Um... This is probably even more complex to set up than other options, and is also almost certainly not free.

And I do keep coming back to the fact that pretty much any direct peer to peer connection requires some degree of technical savy on the part of everyone involved. At the very least whomever is "hosting" the connection has to have some decent abilities to do this. The other folks can act just as clients. But, yeah, that's basically what you are already doing with a lot of the cloud based hosting services anyway. And they have a lot more time and money to make the use of their services as "easy" as possible.