Donnerstag, Februar 27, 2020

Upload GROßER Dateien über HTTP

Ich versuche zum hochladen von wirklich großen VM-Images (5-15 Gb Größe) an einen HTTP-server mit PowerShell.

Habe ich versucht zu verwenden, dass für einige Methoden (hier links zu Skript mit net.WebClient.UploadFile und Skript mit “ Invoke-webRequest)

Es gut funktioniert für Dateien kleiner als 2 GB sind, nicht jedoch für Dateien, die größer als dieser.

Ich versuche, die Arbeit mit httpWebRequest direkt, aber ich nicht in der Lage zu setzen FileStream hinein.

Also meine Frage ist: wie filestream in webrequest?

Oder allgemeiner: wie zum hochladen von großen Dateien über http mit PowerShell?

$Timeout=10000000;
$fileName = "0.iso";
$data = "C:\\$fileName";
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName";
#$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": 
#$buffer = gc -en byte $data # too much space in memory 
$buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb
[System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
#$webRequest.ContentLength = $buffer.Length;
$webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123");

$requestStream = $webRequest.GetRequestStream()
$requestStream.Write($buffer, 0, $buffer.Length)
$requestStream.Flush()
$requestStream.Close()

[System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse()
$streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream())
$result = $streamReader.ReadToEnd()
return $result
$stream.Close() 
  • Sind Sie sicher, dass das nicht eine Einschränkung der server, die Sie hochladen zu? Gibt es eine maximale POST-Größe einstellen?
  • dieser Befehl von linux-box funktioniert gut und einfach hochladen von großen Dateien: curl -v -u admin:admin123 --upload-file file.iso http://nexus.ndlab.local:8081/nexus/content/sites/myproj/fromlinux.iso
InformationsquelleAutor vvchik | 2015-10-20

3 Kommentare

  1. 9

    Standardmäßig HttpWebRequest ist die Pufferung von Daten im Speicher.
    Setzen Sie einfach HttpWebRequest.AllowWriteStreamBuffering-Eigenschaft auf false, und Sie wäre in der Lage, das hochladen von Dateien mit nahezu beliebiger Größe.
    Weitere Einzelheiten finden Sie unter msdn

  2. 9

    Danke @Stoune, es war die Letzte Sache, die geholfen empfangen Sie schließlich funktionierende Lösung.

    Einer mehr, es ist notwendig, sich zu organisieren stream-Datei Lesen und schreiben, um die webrequest-Puffer. Und es möglicherweise zu tun mit diesem Stück code:

    $requestStream = $webRequest.GetRequestStream()
    $fileStream = [System.IO.File]::OpenRead($file)
    $chunk = New-Object byte[] $bufSize
      while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
      {
        $requestStream.write($chunk, 0, $bytesRead)
        $requestStream.Flush()
      }

    Und endgültige Skript wie folgt Aussehen:

    $user = "admin"
    $pass = "admin123"
    $dir = "C:\Virtual Hard Disks"
    $fileName = "win2012r2std.vhdx"
    $file = "$dir/$fileName"
    $url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"
    $Timeout=10000000
    $bufSize=10000
    
    $cred = New-Object System.Net.NetworkCredential($user, $pass)
    
    $webRequest = [System.Net.HttpWebRequest]::Create($url)
    $webRequest.Timeout = $timeout
    $webRequest.Method = "POST"
    $webRequest.ContentType = "application/data"
    $webRequest.AllowWriteStreamBuffering=$false
    $webRequest.SendChunked=$true # needed by previous line
    $webRequest.Credentials = $cred
    
    $requestStream = $webRequest.GetRequestStream()
    $fileStream = [System.IO.File]::OpenRead($file)
    $chunk = New-Object byte[] $bufSize
      while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
      {
        $requestStream.write($chunk, 0, $bytesRead)
        $requestStream.Flush()
      }
    
    $responceStream = $webRequest.getresponse()
    #$status = $webRequest.statuscode
    
    $FileStream.Close()
    $requestStream.Close()
    $responceStream.Close()
    
    $responceStream
    $responceStream.GetResponseHeader("Content-Length") 
    $responceStream.StatusCode
    #$status
  3. 1

    Beim hochladen von Sonatype Nexus3 ich verwendete den folgenden code ein. Hat mich einige Zeit, um es herauszufinden, arbeiten mit hoch-und response von Nexus3 und das hochladen und herunterladen von großen Dateien (größer als 2GB). Wir haben Apache vor Nexus3 kümmert sich um die https-verbindungen.
    Die Verwendung der basic-Authentifizierung sicher, dass die Nexus reagiert hat Recht, wenn Apache davor.
    Senden pre-Authentifizierung über KOPF und mit chunked upload für große Dateien fest, dass das hochladen von großen Dateien nicht vorzeitig beendet wird.

    Herunterladen großer Dateien über Invoke-WebRequest würde scheitern, auch mit unterschiedlichen Fehlern. Jetzt benutze ich WebRequest über .Net und es funktioniert, siehe Download-File.

    Und wenn der code ausgeführt wurde, über einen automatisierten Prozess (von System Center Orchestrator), https scheitern würden. So zwingen wir TLS 1.2 als Schema https erkannt wird.

    function New-HttpWebRequest
    {
        <#
        .SYNOPSIS
        Creates a new [System.Net.HttpWebRequest] ready for file transmission.
    
        .DESCRIPTION
        Creates a new [System.Net.HttpWebRequest] ready for file transmission.
        The method will be Put. If the filesize is larger than the buffersize,
        the HttpWebRequest will be configured for chunked transfer.
    
        .PARAMETER Url
        Url to connect to.
    
        .PARAMETER Credential
        Credential for authentication at the Url resource.
    
        .EXAMPLE
        An example
        #>
        param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$Url,
    
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [pscredential]$Credential,
    
            [Parameter(Mandatory=$true)]
            [long]$FileSize,
    
            [Parameter(Mandatory=$true)]
            [long]$BufferSize
        )
    
        $webRequest = [System.Net.HttpWebRequest]::Create($Url)
        $webRequest.Timeout = 600 * 1000;
        $webRequest.ReadWriteTimeout = 600 * 1000;
        $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11;
        $webRequest.Method = "PUT";
        $webRequest.ContentType = "application/octet-stream";
        $webRequest.KeepAlive = $true;
        $webRequest.UserAgent = "<I use a specific UserAgent>";
        #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)';
        $webRequest.PreAuthenticate = $true;
        $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
        $webRequest.Headers["Authorization"] = "Basic $auth"
    
        if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize)
        {
            Write-Verbose "FileSize is greater than BufferSize, using chunked transfer.";
            $webRequest.AllowWriteStreamBuffering = $false;
            $webRequest.SendChunked = $true;
        }
        else
        {
            # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write.
            # Chunked cannot be used in this case.
            $webRequest.AllowWriteStreamBuffering = $true;
            $webRequest.SendChunked = $false;
            $webRequest.ContentLength = $FileSize;
        }
    
        return $webRequest;
    }
    
    function Get-BufferSize
    {
        <#
        .SYNOPSIS
        Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size.
    
        .DESCRIPTION
        Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default.
    
        .PARAMETER ByteLength
        Length of the bytes for which to calculate a valid buffer size.
    
        .PARAMETER AtLeast
        The minimum required buffer size, default 1MB.
    
        .EXAMPLE
        Get-BufferSize 4283304773
    
        Returns 42991616 which is 41MB.
    
        .EXAMPLE
        Get-BufferSize 4283304
    
        Returns 1048576 which is 1MB.
    
        .EXAMPLE
        Get-BufferSize 4283304 5MB
    
        Returns 5242880 which is 5MB.
        #>
        param(
            [Parameter(Mandatory=$true)]
            [long]$ByteLength,
    
            [long]$AtLeast = 1MB
        )
    
        [long]$size = $ByteLength / 100;
        if ($size -lt $AtLeast)
        {
            $size = $AtLeast;
        }
        else
        {
            $size = [Math]::Round($size / 1MB) * 1MB;
        }
    
        return $size;
    }
    
    function Get-UseChunkedUpload
    {
        param(
            [Parameter(Mandatory=$true)]
            [long]$FileSize,
    
            [Parameter(Mandatory=$true)]
            [long]$BufferSize
        )
    
        return $FileSize -gt $BufferSize;
    }
    
    function Configure-Tls
    {
        param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$Url
        )
    
        [System.Uri]$uri = $Url;
        if ($uri.Scheme -eq "https")
        {
            Write-Verbose "Using TLS 1.2";
            [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
        }
    }
    
    function Send-PreAuthenticate
    {
        param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$Url,
    
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [pscredential]$Credential
        )
    
        $response = $null;
        try
        {
            [System.Uri]$uri = $Url;
            $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/');
            Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority";
            $wr = [System.Net.WebRequest]::Create($repositoryAuthority);
            $wr.Method = "HEAD";
            $wr.PreAuthenticate = $true;
            $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
            $wr.Headers["Authorization"] = "Basic $auth"
            $response = $wr.GetResponse();
        }
        finally
        {
            if ($response)
            {
                $response.Close();
                $response.Dispose();
                $response = $null;
            }
        }
    }
    
    function Upload-File
    {
        <#
        .SYNOPSIS
        Uploads a file to the Nexus repository.
    
        .DESCRIPTION
        Uploads a file to the Nexus repository.
        If the file was uploaded successfully, the url via which the resource can be downloaded is returned.
    
        .PARAMETER Url
        The Url where the resource should be created.
        Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload.
    
        .PARAMETER File
        The file that should be uploaded.
    
        .PARAMETER Credential
        Credential used for authentication at the Nexus repository.
    
        .EXAMPLE
        Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential)
    
        .OUTPUTS
        If the file was uploaded successfully, the url via which the resource can be downloaded.
        #>
        param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$Url,
    
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [System.IO.FileInfo]$File,
    
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [pscredential]$Credential
        )
    
        Write-Verbose "Upload-File Url:$Url"
    
        Configure-Tls -Url $Url;
    
        $fileSizeBytes = $File.Length;
        #$bufSize = Get-BufferSize $fileSizeBytes;
        $bufSize = 4 * 1MB;
        Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB));
        if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize)
        {
            Write-Verbose "Using chunked upload. Send pre-auth first.";
            Send-PreAuthenticate -Url $Url -Credential $Credential;
        }
    
        $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length);
        $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize;
        $chunk = New-Object byte[] $bufSize;
        $bytesWritten = 0;
        $fileStream = [System.IO.File]::OpenRead($File.FullName);
        $requestStream = $WebRequest.GetRequestStream();
        try
        {
            while($bytesRead = $fileStream.Read($chunk,0,$bufSize))
            {
                $requestStream.Write($chunk, 0, $bytesRead);
                $requestStream.Flush();
                $bytesWritten += $bytesRead;
                $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB));
                Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100);
            }
        }
        catch
        {
            throw;
        }
        finally
        {
            if ($fileStream)
            {
                $fileStream.Close();
            }
            if ($requestStream)
            {
                $requestStream.Close();
                $requestStream.Dispose();
                $requestStream = $null;
            }
            Write-Progress -Activity $progressActivityMessage -Completed;
        }
    
        # Read the response.
        $response = $null;
        try
        {
            $response = $webRequest.GetResponse();
            Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri);
            return $response.ResponseUri;
        }
        catch
        {
            if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*"))
            {
                throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.")
            }
    
            throw;
        }
        finally
        {
            if ($response)
            {
                $response.Close();
                $response.Dispose();
                $response = $null;
            }
            if ($webRequest)
            {
                $webRequest = $null;
            }
        }
    }
    
    function Download-File
    {
        param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$Url,
    
            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]$FileName
        )
    
        $SDXDownloadType = @"
        using System.IO;
        using System.Net;
    
        public class SDXDownload
        {
            static public void DownloadFile(string Uri, string Filename)
            {
                HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri);
                webRequest.Method = "GET";
                using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse())
                using (Stream fileStream = File.OpenWrite(Filename))
                using (Stream streamResponse = myHttpWebResponse.GetResponseStream())
                {
                    int bufSize = 64 * 1024;
                    byte[] readBuff = new byte[bufSize];
                    int bytesRead = streamResponse.Read(readBuff, 0, bufSize);
                    while (bytesRead > 0)
                    {
                        fileStream.Write(readBuff, 0, bytesRead);
                        bytesRead = streamResponse.Read(readBuff, 0, 256);
                    }
                }
            }
        }
    "@
    
        Configure-Tls -Url $Url;
    
        Add-Type -TypeDefinition $SDXDownloadType;
        [SDXDownload]::DownloadFile($Url, $FileName);
    }
    • Danke, danke! Ich habe gekämpft, zwei Tage und das ist die einzige Lösung, die mir erlaubt zum bereitstellen von großen Dateien an artifactory. Vergessen Sie nicht, zu aktualisieren $webRequest.UserAgent zukünftige Nutzer.

Kostenlose Online-Tests