Dealing with dangling references to custom smart pointer - c++11

Question

Disclaimer: yes, I know about shared_ptr. But I still want to do this.
Also, I am aware I am not using locks nor atomics and so this is not thread safe.
Assume the following somewhat simple implementation of a 'smart' pointer. The semantics being that if you pass by value you increase the ref count and if you stick to references you have a 'weak' of sorts:
#pragma once
#include <iostream>
#include <cassert>
using std::cout;
template <class T>
class Ptr {
private:
T* ptr;
int* refCount;
public:
Ptr(T* ptr) {
assert(ptr);
this->ptr = ptr;
this->refCount = new int(1);
}
Ptr(Ptr& from) {
swap(*this, from);
}
Ptr(Ptr&& from) {
swap(*this, from, false);
}
Ptr& operator=(const Ptr& from) {
swap(*this, from);
return *this;
}
~Ptr() {
if (*refCount >= 1) {
(*refCount)--;
cout << "\n Ptr destructor: new ref count: " << *refCount;
if (*refCount == 0) {
delete ptr;
ptr = nullptr;
}
}
}
operator bool() {
return (*refCount >= 1);
}
T* operator->() {
assert(*refCount >= 1);
return ptr;
}
int referenceCount() {
return *refCount;
}
private:
template <class T>
void swap(Ptr<T>& to, Ptr<T>& from, bool isCopy = true) {
assert((*from.refCount) >= 1);
to.ptr = from.ptr;
to.refCount = from.refCount;
if (isCopy) {
(*to.refCount)++;
}
else {
from.ptr = nullptr;
}
}
};
class A {
public:
A() = default;
~A() {
cout << "\n dealloc:" << this;
}
void doSomething() {
}
};
void Test() {
{
Ptr<A> p1(new A);
cout << "\ncheckpoint - refCount (should be 1): " << p1.referenceCount();
Ptr<A> p2 = p1;
cout << "\ncheckpoint - refCount (should be 2): " << p1.referenceCount();
Ptr<A>& p3 = p1;
cout << "\ncheckpoint - refCount (should be 2): " << p1.referenceCount();
Ptr<A> p4 = std::move(p1);
cout << "\ncheckpoint - refCount (should be 2): " << p4.referenceCount();
Ptr<A> p5 = p4;
cout << "\ncheckpoint - refCount (should be 3): " << p5.referenceCount();
}
cout << "\nend";
}
I get the following output which is as expected:
checkpoint - refCount (should be 1): 1
checkpoint - refCount (should be 2): 2
checkpoint - refCount (should be 2): 2
checkpoint - refCount (should be 2): 2
checkpoint - refCount (should be 3): 3
Ptr destructor: new ref count: 2
Ptr destructor: new ref count: 1
Ptr destructor: new ref count: 0
dealloc:00C3D060
But there is one big problem.
Assume I had kept any of those references (could have been a pointer for that matter -- any handle that acts as a weak ref), and the last strong ref goes out of scope. The original pointer is now null and the smart pointer's counter (which I am leaking), would still allow any weak handles to behave appropriately since it would still hold a valid value of 0.
But had I deleted the counter pointer, any references left would eventually point to other data if that memory location got used for something else, which could include any number other than 0.
So I'm thinking I might need to crate a static pointer to a value of 0 and instead use pointers to pointers for the counter? That way whichever strong ref deletes the pointer could also point all counters to the static 0?
I'm kind of confused about it. Especially since I was looking at the source code of a rendering engine and their shared ptr class is very similar to the above code.