Customer-controlled storage account recovery (CSAR) is now live in Azure Portal

Updated 2021-6-26
Recover a deleted storage account
https://docs.microsoft.com/en-us/azure/storage/common/storage-account-recover

===============================================

How to recover a deleted Azure Storage Account:

  1. Navigate to the ‘New support request’ workflow

  2. Select the appropriate options on the ‘Basics’ tab (see screenshot below) and navigate to the ‘Solutions’ tab

  3. Click the ‘Recover a deleted storage account’ blade link on top of the solutions tab

  4. Select the deleted storage account you would like to recover from the dropdown and initiate the recovery process by clicking the button in the command bar at the bottom of the blade.

HTH. 2020-10-08 By Jacky

Prefetch Azure Service Bus messages is faster?

Prefetch speeds up the message flow by having a message readily available for local retrieval when and before the application asks for one. This throughput gain is the result of a trade-off that the application author must make explicitly:

With the ReceiveAndDelete receive mode, all messages that are acquired into the prefetch buffer are no longer available in the queue, and only reside in the in-memory prefetch buffer until they are received into the application through the Receive/ReceiveAsync or OnMessage/OnMessageAsync APIs. If the application terminates before the messages are received into the application, those messages are irrecoverably lost.

In the PeekLock receive mode, messages fetched into the Prefetch buffer are acquired into the buffer in a locked state, and have the timeout clock for the lock ticking. If the prefetch buffer is large, and processing takes so long that message locks expire while residing in the prefetch buffer or even while the application is processing the message, there might be some confusing events for the application to handle.

The application might acquire a message with an expired or imminently expiring lock. If so, the application might process the message, but then find that it cannot complete it due to a lock expiration. The application can check the LockedUntilUtc property (which is subject to clock skew between the broker and local machine clock). If the message lock has expired, the application must ignore the message; no API call on or with the message should be made. If the message is not expired but expiration is imminent, the lock can be renewed and extended by another default lock period by calling message.RenewLock()

If the lock silently expires in the prefetch buffer, the message is treated as abandoned and is again made available for retrieval from the queue. That might cause it to be fetched into the prefetch buffer; placed at the end. If the prefetch buffer cannot usually be worked through during the message expiration, this causes messages to be repeatedly prefetched but never effectively delivered in a usable (validly locked) state, and are eventually moved to the dead-letter queue once the maximum delivery count is exceeded.

If you need a high degree of reliability for message processing, and processing takes significant work and time, it is recommended that you use the prefetch feature conservatively, or not at all.

If you need high throughput and message processing is commonly cheap, prefetch yields significant throughput benefits.

The maximum prefetch count and the lock duration configured on the queue or subscription need to be balanced such that the lock timeout at least exceeds the cumulative expected message processing time for the maximum size of the prefetch buffer, plus one message. At the same time, the lock timeout ought not to be so long that messages can exceed their maximum TimeToLive when they are accidentally dropped, thus requiring their lock to expire before being redelivered.

For more information:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-prefetch

Sample Code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ServiceBusPrefetchTest
{
    class Program
    {
        static void Main(string[] args)
		{
			var sender = ServiceBusClientFactory.CreateTopicClient();
			var receiver = ServiceBusClientFactory.CreateMessageReceiver();
			var bufferSize = 2_000_0;
			var messageCount = 100;

			// ************************************************
			// Change the MaxMessageCount and PrefetchCount, and check the Total processing time.
			// Receive messages with/without prefetch.
			// receiver.PrefetchCount = 0; // Disable prefetch
			receiver.PrefetchCount = 20;
			var maxMsgCount = 15;
			Console.WriteLine("receiver.PrefetchCount = {0}", receiver.PrefetchCount);
			Console.WriteLine("maxMsgCount = {0}", maxMsgCount);
			// *************************************************

			// Gerenarte and send random buffers.
			for (var i = 0; i < messageCount; i++)
			{
				var buffer = GenerateRandomBuffer(bufferSize);
				var message = new Message(buffer);
				message.Body = buffer;
				sender.SendAsync(message).GetAwaiter().GetResult();
			}

			var messageList = new List<Message>();
			var stopwatch = new Stopwatch();
			stopwatch.Start();

			while (messageList.Count < messageCount)
			{	
				var messageToAdd = receiver.ReceiveAsync(maxMsgCount, TimeSpan.FromMilliseconds(1000)).GetAwaiter()
				.GetResult()?.OrderBy(message => message.SystemProperties.SequenceNumber).ToList();

				if (messageToAdd == null || messageToAdd.Count == 0)
				{
					Console.WriteLine("Received 0 messages");
					continue;
				}

				// Simulate other process time, such as saving to databse.
				Thread.Sleep(10);

				Console.WriteLine($"Get message:{messageToAdd.Count}");
				messageList.AddRange(messageToAdd);
				receiver.CompleteAsync(messageToAdd.Select(m => m.SystemProperties.LockToken));
			}
			stopwatch.Stop();
			var elapsed_time = stopwatch.ElapsedMilliseconds;
			Console.WriteLine($"Total processing time: {elapsed_time} ms");
			Console.WriteLine($"Total message count: Expect: {messageCount}, Get: {messageList.Count}");
		}

		// Define other methods and classes here
		public static class ServiceBusClientFactory
		{
			public static string ServiceBusConnectionString { get; } = "Your ConnectionString";
			public static string TopicName { get; } = "Your Topic Name";
			public static string SubscriptionNameForDbSettlement { get; } = "Your Subscription Name";

			public static ITopicClient CreateTopicClient()
			{
				return new TopicClient(ServiceBusConnectionString, TopicName);
			}

			public static IMessageReceiver CreateMessageReceiver()
			{
				return new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(TopicName, SubscriptionNameForDbSettlement), ReceiveMode.PeekLock);
			}
		}

		public static byte[] GenerateRandomBuffer(long size)
		{
			Random random = new Random();
			var bytes = new byte[size];
			random.NextBytes(bytes);
			return bytes;
		}
	}
}

Test result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
PS C:\Jacky\ServiceBusPrefetchTest\bin\Debug> .\ServiceBusPrefetchTest.exe
receiver.PrefetchCount = 0
var maxMsgCount = 5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Get message:5
Total processing time: 943 ms
Total message count: Expect: 100, Get: 100

PS C:\Jacky\ServiceBusPrefetchTest\bin\Debug> .\ServiceBusPrefetchTest.exe
receiver.PrefetchCount = 20
var maxMsgCount = 15
Get message:15
Get message:5
Get message:13
Get message:13
Get message:13
Get message:13
Get message:13
Get message:13
Get message:13
Total processing time: 1134 ms
Total message count: Expect: 100, Get: 111

HTH. 2020-9-29 By Jacky

Occasionally Receive Messages in Unexpected Sequence

Below sample code can reproduce it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ServiceBusTestClient
{
	class Program
	{
		static void Main()
		{
			// Create sender and receiver
			var sender = ServiceBusClientFactory.CreateTopicClient();
			var receiver = ServiceBusClientFactory.CreateMessageReceiver();

			// Please change this value according to your service bus settings.
			long lastSeq = 82354;

			var rand = new Random();
			while (true)
			{
				// Sender Loop
				var senderSize = rand.Next(1, 10);
				for (var i = 0; i < senderSize; i++)
				{
					// Send one message
					var message = new Message();
					message.Body = Encoding.UTF8.GetBytes($"Test message body {DateTime.UtcNow}");
					sender.SendAsync(message).GetAwaiter().GetResult();
				}

				//Receiver Loop				
				var result = receiver.ReceiveAsync(5, TimeSpan.FromMilliseconds(5000)).GetAwaiter().
                GetResult()?.OrderBy(message => message.SystemProperties.SequenceNumber).ToList();

                var seqNumbers = result.Select(item => item.SystemProperties.SequenceNumber).ToList();

				if (CheckMissingSequenceNumberExists(lastSeq + 1, seqNumbers))
                {
                    throw new Exception($"Missing Sequence Number. Expect: {lastSeq + 1}, Get: {string.Join(",", seqNumbers)}");
                }

                lastSeq = seqNumbers[seqNumbers.Count - 1];
                var lockTokens = result.Select(item => item.SystemProperties.LockToken).ToList();
                receiver.CompleteAsync(lockTokens).GetAwaiter().GetResult();
            }
		}

		// Define other methods and classes here
		public static bool CheckMissingSequenceNumberExists(long initialNum, List<long> dequeuedSequenceNumbers)
		{
			Console.WriteLine("initialNum:{0}, dequeuedSequenceNumbers:{1}", initialNum, dequeuedSequenceNumbers[0]);

			if (dequeuedSequenceNumbers[0] != initialNum)
			{
				return true;
			}

			//Check sequence number is increased by 1, and if not, set the flag as true.
			for (int i = 1; i < dequeuedSequenceNumbers.Count; i++)
			{
				if (dequeuedSequenceNumbers[i] - dequeuedSequenceNumbers[i - 1] != 1)
				{
					return true;
				}
			}

			return false;
		}

		public static class ServiceBusClientFactory
		{
			public static string ServiceBusConnectionString { get; } = "Your connection String";
			public static string TopicName { get; } = "Your Topic Name";
			public static string SubscriptionNameForDbSettlement { get; } = "Your Subscrption Name";

			public static ITopicClient CreateTopicClient()
			{
				return new TopicClient(ServiceBusConnectionString, TopicName);
			}

			public static IMessageReceiver CreateMessageReceiver()
			{
				return new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(TopicName, SubscriptionNameForDbSettlement), ReceiveMode.PeekLock);
			}
		}
	}
}

Below sample code can prevent it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ServiceBusTestClient
{
	class Program
	{
		static void Main()
		{
			// Create sender and receiver
			var sender = ServiceBusClientFactory.CreateTopicClient();
			var receiver = ServiceBusClientFactory.CreateMessageReceiver();

			// Please change this value according to your service bus settings.
			long lastSeq = 82354;

			var rand = new Random();
			while (true)
			{
				// Sender Loop
				var senderSize = rand.Next(1, 10);
				for (var i = 0; i < senderSize; i++)
				{
					// Send one message
					var message = new Message();
					message.Body = Encoding.UTF8.GetBytes($"Test message body {DateTime.UtcNow}");
					sender.SendAsync(message).GetAwaiter().GetResult();
				}

				// *******************************************************************************
				// With .NET, you enable the Prefetch feature by setting the PrefetchCount property of a MessageReceiver, QueueClient, or SubscriptionClient to a number greater than zero. Setting the value to zero turns off prefetch.

				//You can easily add this setting to the receive-side of the QueuesGettingStarted or ReceiveLoop samples' settings to see the effect in those contexts.

				//While messages are available in the prefetch buffer, any subsequent Receive / ReceiveAsync calls are immediately fulfilled from the buffer, and the buffer is replenished in the background as space becomes available. If there are no messages available for delivery, the receive operation empties the buffer and then waits or blocks, as expected.
				receiver.PrefetchCount = 5;
				// https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-prefetch
				// ********************************************************************************
				//Receiver Loop				
				var result = receiver.ReceiveAsync(5, TimeSpan.FromMilliseconds(5000)).GetAwaiter().
                GetResult()?.OrderBy(message => message.SystemProperties.SequenceNumber).ToList();

                var seqNumbers = result.Select(item => item.SystemProperties.SequenceNumber).ToList();

				if (CheckMissingSequenceNumberExists(lastSeq + 1, seqNumbers))
                {
                    throw new Exception($"Missing Sequence Number. Expect: {lastSeq + 1}, Get: {string.Join(",", seqNumbers)}");
                }

                lastSeq = seqNumbers[seqNumbers.Count - 1];
                var lockTokens = result.Select(item => item.SystemProperties.LockToken).ToList();
                receiver.CompleteAsync(lockTokens).GetAwaiter().GetResult();
            }
		}

		// Define other methods and classes here
		public static bool CheckMissingSequenceNumberExists(long initialNum, List<long> dequeuedSequenceNumbers)
		{
			Console.WriteLine("initialNum:{0}, dequeuedSequenceNumbers:{1}", initialNum, dequeuedSequenceNumbers[0]);

			if (dequeuedSequenceNumbers[0] != initialNum)
			{
				return true;
			}

			//Check sequence number is increased by 1, and if not, set the flag as true.
			for (int i = 1; i < dequeuedSequenceNumbers.Count; i++)
			{
				if (dequeuedSequenceNumbers[i] - dequeuedSequenceNumbers[i - 1] != 1)
				{
					return true;
				}
			}

			return false;
		}

		public static class ServiceBusClientFactory
		{
			public static string ServiceBusConnectionString { get; } = "Your connection String";
			public static string TopicName { get; } = "Your Topic Name";
			public static string SubscriptionNameForDbSettlement { get; } = "Your Subscrption Name";

			public static ITopicClient CreateTopicClient()
			{
				return new TopicClient(ServiceBusConnectionString, TopicName);
			}

			public static IMessageReceiver CreateMessageReceiver()
			{
				return new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(TopicName, SubscriptionNameForDbSettlement), ReceiveMode.PeekLock);
			}
		}
	}
}

Collect X-Forward-For in Azure Web App

  1. Create a web application and config Application Insights.

  2. Telemetry initializer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;

namespace TestAppClientIp
{
    public class CloneIPAddress : ITelemetryInitializer
    {
        public void Initialize(ITelemetry telemetry)
        {
            ISupportProperties propTelemetry = telemetry as ISupportProperties;

            if (propTelemetry != null && !propTelemetry.Properties.ContainsKey("client-ip"))
            {
                string clientIPValue = telemetry.Context.Location.Ip;
                propTelemetry.Properties.Add("client-ip", clientIPValue);
            }
        }
    }
}
  1. Enable telemetry initializer for ASP.NET
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
using Microsoft.ApplicationInsights.Extensibility;

using System;
using System.Web;
using System.Web.Optimization;
using System.Web.Routing;

namespace TestAppClientIp
{
    public class Global : HttpApplication
    {
        void Application_Start(object sender, EventArgs e)
        {
            // Code that runs on application startup
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);

            //Enable your telemetry initializer:
            TelemetryConfiguration.Active.TelemetryInitializers.Add(new CloneIPAddress());
        }
    }
}
  1. Sending sample HTTP requests with X-Forwarded-For header or your custom header
1
Invoke-WebRequest "https://jackywin3.azurewebsites.net/About" -Method GET -Headers @{"X-Forwarded-For" = "1.1.0.1"}
  1. View the results of your telemetry initializer
    If you send new traffic to your site, and wait a few minutes. You can then run a query to confirm collection is working:
    requests
    | where timestamp > ago(1h)
    | project appName, operation_Name, url, resultCode, client_IP, customDimensions.[“client-ip”]

Application Insights ClientIP

For more information: https://docs.microsoft.com/en-us/azure/azure-monitor/app/ip-collection?tabs=net

Enjoy. 2020-9-23 By Jacky

Multiple issuers and openid-config API Management Service 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<policies>
    <inbound>
        <validate-jwt header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized. Access token is missing or invalid.">
            <openid-config url="https://login.microsoftonline.com/XXXXXXXXX/.well-known/openid-configuration" />
            <openid-config url="https://login.microsoftonline.com/BBBBBBBBB/.well-known/openid-configuration" />
            <audiences>
                <audience>44444444-0000-0000-0000-444444444444</audience>
                <audience>66666666-0000-0000-0000-666666666666</audience>
            </audiences>
            <issuers>
                <issuer>https://login.microsoftonline.com/99999999-0000-0000-0000-999999999999/v2.0/</issuer>
                <issuer>https:jacky.b2clogin.com/99999999-0000-0000-0000-999999999999/v2.0/</issuer>
            </issuers>
        </validate-jwt>
        <base />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

More Information: https://docs.microsoft.com/en-us/azure/api-management/api-management-access-restriction-policies#ValidateJWT

Enjoy. 2020-9-23 By Jacky

New Azure SQL Learning Tools - 2020 August

New Azure SQL Learning Tools help reduce the global technology skills gap

https://aka.ms/learnazuresql

  1. Azure SQL Fundamentals (aka.ms/azuresqlfundamentals) – a 6-course learning path on Microsoft Learn. This includes built-in sandbox environments and subscriptions

  2. Azure SQL for beginners (aka.ms/azuresql4beginners) – a 61-video series which walks through all the content on the learning path and more.

  3. Azure SQL Workshop (aka.ms/asqlworkshop, then select Workshop: Azure SQL) – a 1-2 day workshop including hands-on labs and slides which are free/open source for everyone to leverage.

  4. Azure SQL Bootcamp (aka.ms/azuresqlbootcamp) –4-day (2 hours each day) live bootcamp from August 17-20.

Enjoy. 2020-Aug-21 by Jacky Chiou

Convert etl to pcapng

** Step 1. Install-Module PowerShellGet -Force **
PowerShell => Run a Administrator

1
2
3
4
5
6
7
8
9
10
PS D:\Users\jacky> Install-Module PowerShellGet -Force

NuGet provider is required to continue
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
 provider must be available in 'D:\Program Files\PackageManagement\ProviderAssemblies' or
'D:\Users\jacky\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by running
 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install and
import the NuGet provider now?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y
PS D:\Users\jacky>

** Step2. Install-Module Convert-Etl2Pcapng -Force -AcceptLicense **
Close PowerShell
PowerShell Run a Administrator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
PS D:\Users\jacky> Install-Module Convert-Etl2Pcapng -Force -AcceptLicense
PS D:\Users\jacky> Get-Module -ListAvailable Convert-Etl2Pcapng


    Directory: D:\Program Files\WindowsPowerShell\Modules


ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2020.5.14  Convert-Etl2Pcapng                  {Register-Etl2Pcapng, Unregister-Etl2Pcapng, Convert-Etl2P...


PS D:\Users\jacky> Register-Etl2Pcapng
PS D:\Users\jacky>

** Step3. Convert Etl to Pcapng **

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
PS D:\demo> netsh trace start capture=yes tracefile=D:\demo\trace.etl report=dis
File "D:\demo\trace.etl" already exists.
One or more parameters for the command are not correct or missing.
See 'trace start help' for detailed help.

PS D:\demo> netsh trace start capture=yes tracefile=D:\demo\trace.etl report=dis

Trace configuration:
-------------------------------------------------------------------
Status:             Running
Trace File:         D:\demo\trace.etl
Append:             Off
Circular:           On
Max Size:           250 MB
Report:             Disabled

PS D:\demo> Start-Sleep 1
PS D:\demo> Clear-DnsClientCache
PS D:\demo> ping cloudflare.com

Pinging cloudflare.com [104.17.176.85] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 104.17.176.85:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
PS D:\demo> Start-Sleep 3
PS D:\demo> netsh trace stop
Correlating traces ... done
Merging traces ... done
File location = D:\demo\trace.etl
Tracing session was successfully stopped.

PS D:\demo>

Convert Etl 2 pcapng

Convert Etl 2 pcapng completed

open it in Wireshark

HTH. 2020-July-20 By Jacky